Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 20:08:00.558383 14318 test_util.cc:268] Using random seed: 186322179
I20240501 20:08:00.569607 14318 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:00.569687 14318 ts_itest-base.cc:113] --------------
I20240501 20:08:00.569733 14318 ts_itest-base.cc:114] 5 tablet servers
I20240501 20:08:00.569802 14318 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:08:00.569873 14318 ts_itest-base.cc:116] --------------
2024-05-01T20:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:00Z Disabled control of system clock
I20240501 20:08:00.594228 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.251.190:34609
--webserver_interface=127.13.251.190
--webserver_port=0
--builtin_ntp_servers=127.13.251.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.251.190:34609 with env {}
W20240501 20:08:00.911690 14330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:00.912019 14330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:00.912156 14330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:00.919488 14330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:00.919637 14330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:00.919718 14330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:00.919840 14330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:00.928678 14330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.251.190:34609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.251.190:34609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.251.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:00.929962 14330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:00.931630 14330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:00.942229 14335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:00.942714 14338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:00.943291 14336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:00.943850 14330 server_base.cc:1034] running on GCE node
I20240501 20:08:00.944598 14330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:00.945988 14330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:00.947222 14330 hybrid_clock.cc:648] HybridClock initialized: now 1714594080947167 us; error 45 us; skew 500 ppm
I20240501 20:08:00.947587 14330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:00.949983 14330 webserver.cc:460] Webserver started at http://127.13.251.190:43709/ using document root <none> and password file <none>
I20240501 20:08:00.950688 14330 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:00.950827 14330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:00.951133 14330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:00.953755 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "602820a1396841b6a39ad944d7befab7"
format_stamp: "Formatted at 2024-05-01 20:08:00 on dist-test-slave-qswb"
I20240501 20:08:00.954499 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "602820a1396841b6a39ad944d7befab7"
format_stamp: "Formatted at 2024-05-01 20:08:00 on dist-test-slave-qswb"
I20240501 20:08:00.959478 14330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 20:08:00.962682 14344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:00.964028 14330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:00.964196 14330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
uuid: "602820a1396841b6a39ad944d7befab7"
format_stamp: "Formatted at 2024-05-01 20:08:00 on dist-test-slave-qswb"
I20240501 20:08:00.964375 14330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:00.982661 14330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:00.983578 14330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:00.983873 14330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:01.001832 14330 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.190:34609
I20240501 20:08:01.001842 14395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.190:34609 every 8 connection(s)
I20240501 20:08:01.003243 14330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:01.007025 14396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:01.009912 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 14330
I20240501 20:08:01.010164 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:01.016028 14396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7: Bootstrap starting.
I20240501 20:08:01.019068 14396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:01.020056 14396 log.cc:824] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:01.022744 14396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7: No bootstrap required, opened a new log
I20240501 20:08:01.031600 14396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602820a1396841b6a39ad944d7befab7" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 34609 } }
I20240501 20:08:01.031939 14396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:01.032043 14396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602820a1396841b6a39ad944d7befab7, State: Initialized, Role: FOLLOWER
I20240501 20:08:01.032660 14396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602820a1396841b6a39ad944d7befab7" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 34609 } }
I20240501 20:08:01.032867 14396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:01.033018 14396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:01.033218 14396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:01.035867 14396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602820a1396841b6a39ad944d7befab7" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 34609 } }
I20240501 20:08:01.036456 14396 leader_election.cc:302] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 602820a1396841b6a39ad944d7befab7; no voters:
I20240501 20:08:01.037101 14396 leader_election.cc:288] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:01.037251 14401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:01.039604 14401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [term 1 LEADER]: Becoming Leader. State: Replica: 602820a1396841b6a39ad944d7befab7, State: Running, Role: LEADER
I20240501 20:08:01.040184 14401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602820a1396841b6a39ad944d7befab7" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 34609 } }
I20240501 20:08:01.039629 14396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:01.043731 14402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "602820a1396841b6a39ad944d7befab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602820a1396841b6a39ad944d7befab7" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 34609 } } }
I20240501 20:08:01.044112 14402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:01.044092 14403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 602820a1396841b6a39ad944d7befab7. Latest consensus state: current_term: 1 leader_uuid: "602820a1396841b6a39ad944d7befab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602820a1396841b6a39ad944d7befab7" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 34609 } } }
I20240501 20:08:01.044468 14403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:01.045881 14409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:01.050987 14409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:01.061311 14409 catalog_manager.cc:1347] Generated new cluster ID: f6163456281342699acd3c28a7299101
I20240501 20:08:01.061434 14409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:01.084075 14409 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:01.085330 14409 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:01.096287 14409 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 602820a1396841b6a39ad944d7befab7: Generated new TSK 0
I20240501 20:08:01.097138 14409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:01.104995 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.129:0
--local_ip_for_outbound_sockets=127.13.251.129
--webserver_interface=127.13.251.129
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--builtin_ntp_servers=127.13.251.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 20:08:01.455374 14420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:01.455706 14420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:01.455814 14420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:01.463304 14420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:01.463492 14420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.129
I20240501 20:08:01.473081 14420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.251.129
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.129
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:01.474532 14420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:01.476675 14420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:01.489516 14425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:01.489562 14426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:01.490060 14428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:01.490706 14420 server_base.cc:1034] running on GCE node
I20240501 20:08:01.491329 14420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:01.492385 14420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:01.493602 14420 hybrid_clock.cc:648] HybridClock initialized: now 1714594081493538 us; error 43 us; skew 500 ppm
I20240501 20:08:01.493921 14420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:01.500792 14420 webserver.cc:460] Webserver started at http://127.13.251.129:41703/ using document root <none> and password file <none>
I20240501 20:08:01.501647 14420 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:01.501816 14420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:01.502152 14420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:01.505132 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cd294148300c4c3a9fb856d7e2f4121b"
format_stamp: "Formatted at 2024-05-01 20:08:01 on dist-test-slave-qswb"
I20240501 20:08:01.506462 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cd294148300c4c3a9fb856d7e2f4121b"
format_stamp: "Formatted at 2024-05-01 20:08:01 on dist-test-slave-qswb"
I20240501 20:08:01.512060 14420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20240501 20:08:01.515731 14434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:01.517382 14420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:01.517570 14420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cd294148300c4c3a9fb856d7e2f4121b"
format_stamp: "Formatted at 2024-05-01 20:08:01 on dist-test-slave-qswb"
I20240501 20:08:01.517761 14420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:01.535001 14420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:01.536079 14420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:01.536417 14420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:01.537585 14420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:01.539196 14420 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:01.539268 14420 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:01.539407 14420 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:01.539484 14420 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:01.574095 14420 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.129:34563
I20240501 20:08:01.574116 14546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.129:34563 every 8 connection(s)
I20240501 20:08:01.576459 14420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:01.584177 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 14420
I20240501 20:08:01.584386 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:01.588449 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.130:0
--local_ip_for_outbound_sockets=127.13.251.130
--webserver_interface=127.13.251.130
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--builtin_ntp_servers=127.13.251.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 20:08:01.591612 14547 heartbeater.cc:347] Connected to a master server at 127.13.251.190:34609
I20240501 20:08:01.591948 14547 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:01.592842 14547 heartbeater.cc:511] Master 127.13.251.190:34609 requested a full tablet report, sending...
I20240501 20:08:01.594846 14361 ts_manager.cc:194] Registered new tserver with Master: cd294148300c4c3a9fb856d7e2f4121b (127.13.251.129:34563)
I20240501 20:08:01.596709 14361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.129:35637
W20240501 20:08:01.953823 14551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:01.954181 14551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:01.954344 14551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:01.963219 14551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:01.963441 14551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.130
I20240501 20:08:01.973161 14551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.251.130
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.130
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:01.974555 14551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:01.976337 14551 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:01.988399 14556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:01.989085 14559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:01.989565 14557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:01.990178 14551 server_base.cc:1034] running on GCE node
I20240501 20:08:01.990757 14551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:01.991731 14551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:01.992960 14551 hybrid_clock.cc:648] HybridClock initialized: now 1714594081992886 us; error 46 us; skew 500 ppm
I20240501 20:08:01.993283 14551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:01.995718 14551 webserver.cc:460] Webserver started at http://127.13.251.130:36829/ using document root <none> and password file <none>
I20240501 20:08:01.996536 14551 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:01.996697 14551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:01.997033 14551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:01.999840 14551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1bc51c3f33d141e79c9cc3db868cedfc"
format_stamp: "Formatted at 2024-05-01 20:08:01 on dist-test-slave-qswb"
I20240501 20:08:02.000576 14551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1bc51c3f33d141e79c9cc3db868cedfc"
format_stamp: "Formatted at 2024-05-01 20:08:01 on dist-test-slave-qswb"
I20240501 20:08:02.006299 14551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:02.009956 14565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:02.012436 14551 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20240501 20:08:02.012763 14551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1bc51c3f33d141e79c9cc3db868cedfc"
format_stamp: "Formatted at 2024-05-01 20:08:01 on dist-test-slave-qswb"
I20240501 20:08:02.013136 14551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:02.032037 14551 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:02.032902 14551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:02.033202 14551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:02.034219 14551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:02.035845 14551 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:02.035946 14551 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:02.036088 14551 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:02.036144 14551 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:02.068791 14551 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.130:37949
I20240501 20:08:02.068812 14678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.130:37949 every 8 connection(s)
I20240501 20:08:02.070227 14551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:02.078902 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 14551
I20240501 20:08:02.079116 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:02.083323 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.131:0
--local_ip_for_outbound_sockets=127.13.251.131
--webserver_interface=127.13.251.131
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--builtin_ntp_servers=127.13.251.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 20:08:02.084616 14679 heartbeater.cc:347] Connected to a master server at 127.13.251.190:34609
I20240501 20:08:02.085037 14679 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:02.086319 14679 heartbeater.cc:511] Master 127.13.251.190:34609 requested a full tablet report, sending...
I20240501 20:08:02.088207 14361 ts_manager.cc:194] Registered new tserver with Master: 1bc51c3f33d141e79c9cc3db868cedfc (127.13.251.130:37949)
I20240501 20:08:02.088999 14361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.130:54921
W20240501 20:08:02.433261 14682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:02.433588 14682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:02.433733 14682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:02.441170 14682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:02.441382 14682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.131
I20240501 20:08:02.450867 14682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.251.131
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.131
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:02.452258 14682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:02.453996 14682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:02.466292 14687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:02.466419 14688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:02.466907 14690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:02.467730 14682 server_base.cc:1034] running on GCE node
I20240501 20:08:02.468326 14682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:02.469285 14682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:02.470497 14682 hybrid_clock.cc:648] HybridClock initialized: now 1714594082470424 us; error 55 us; skew 500 ppm
I20240501 20:08:02.470790 14682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:02.473063 14682 webserver.cc:460] Webserver started at http://127.13.251.131:33489/ using document root <none> and password file <none>
I20240501 20:08:02.473762 14682 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:02.473896 14682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:02.474186 14682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:02.476775 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb"
format_stamp: "Formatted at 2024-05-01 20:08:02 on dist-test-slave-qswb"
I20240501 20:08:02.477753 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb"
format_stamp: "Formatted at 2024-05-01 20:08:02 on dist-test-slave-qswb"
I20240501 20:08:02.482919 14682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 20:08:02.486361 14696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:02.487679 14682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:02.487819 14682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb"
format_stamp: "Formatted at 2024-05-01 20:08:02 on dist-test-slave-qswb"
I20240501 20:08:02.488006 14682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:02.509066 14682 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:02.510083 14682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:02.510380 14682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:02.511344 14682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:02.512936 14682 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:02.513015 14682 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:02.513160 14682 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:02.513217 14682 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:02.546452 14682 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.131:41767
I20240501 20:08:02.546487 14809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.131:41767 every 8 connection(s)
I20240501 20:08:02.547886 14682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:02.553223 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 14682
I20240501 20:08:02.553453 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:02.556881 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.132:0
--local_ip_for_outbound_sockets=127.13.251.132
--webserver_interface=127.13.251.132
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--builtin_ntp_servers=127.13.251.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 20:08:02.562436 14810 heartbeater.cc:347] Connected to a master server at 127.13.251.190:34609
I20240501 20:08:02.562808 14810 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:02.564028 14810 heartbeater.cc:511] Master 127.13.251.190:34609 requested a full tablet report, sending...
I20240501 20:08:02.565809 14361 ts_manager.cc:194] Registered new tserver with Master: 0e7f96e0745646ec9f5bc9d3f3aeb8eb (127.13.251.131:41767)
I20240501 20:08:02.566633 14361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.131:60419
I20240501 20:08:02.599604 14547 heartbeater.cc:503] Master 127.13.251.190:34609 was elected leader, sending a full tablet report...
W20240501 20:08:02.881800 14813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:02.882119 14813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:02.882242 14813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:02.889765 14813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:02.889997 14813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.132
I20240501 20:08:02.899106 14813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.13.251.132
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.132
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:02.900501 14813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:02.902243 14813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:02.913753 14818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:02.914484 14819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:02.915114 14821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:02.915720 14813 server_base.cc:1034] running on GCE node
I20240501 20:08:02.916335 14813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:02.917271 14813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:02.918500 14813 hybrid_clock.cc:648] HybridClock initialized: now 1714594082918442 us; error 34 us; skew 500 ppm
I20240501 20:08:02.918812 14813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:02.921164 14813 webserver.cc:460] Webserver started at http://127.13.251.132:37941/ using document root <none> and password file <none>
I20240501 20:08:02.921948 14813 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:02.922071 14813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:02.922410 14813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:02.925097 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b863e2a9eb8440d1a876b4e67e49aace"
format_stamp: "Formatted at 2024-05-01 20:08:02 on dist-test-slave-qswb"
I20240501 20:08:02.925808 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b863e2a9eb8440d1a876b4e67e49aace"
format_stamp: "Formatted at 2024-05-01 20:08:02 on dist-test-slave-qswb"
I20240501 20:08:02.930859 14813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:08:02.934194 14827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:02.935475 14813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:02.935647 14813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b863e2a9eb8440d1a876b4e67e49aace"
format_stamp: "Formatted at 2024-05-01 20:08:02 on dist-test-slave-qswb"
I20240501 20:08:02.935822 14813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:02.953529 14813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:02.954416 14813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:02.954704 14813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:02.955760 14813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:02.957238 14813 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:02.957324 14813 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:02.957460 14813 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:02.957516 14813 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:02.987831 14813 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.132:36507
I20240501 20:08:02.987861 14941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.132:36507 every 8 connection(s)
I20240501 20:08:02.989107 14813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:02.995867 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 14813
I20240501 20:08:02.996023 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:02.999228 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.133:0
--local_ip_for_outbound_sockets=127.13.251.133
--webserver_interface=127.13.251.133
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--builtin_ntp_servers=127.13.251.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 20:08:03.001606 14942 heartbeater.cc:347] Connected to a master server at 127.13.251.190:34609
I20240501 20:08:03.001914 14942 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:03.002825 14942 heartbeater.cc:511] Master 127.13.251.190:34609 requested a full tablet report, sending...
I20240501 20:08:03.004367 14361 ts_manager.cc:194] Registered new tserver with Master: b863e2a9eb8440d1a876b4e67e49aace (127.13.251.132:36507)
I20240501 20:08:03.005545 14361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.132:59001
I20240501 20:08:03.091873 14679 heartbeater.cc:503] Master 127.13.251.190:34609 was elected leader, sending a full tablet report...
W20240501 20:08:03.330466 14945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:03.330761 14945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:03.330888 14945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:03.338217 14945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:03.338428 14945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.133
I20240501 20:08:03.347909 14945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.13.251.133
--webserver_port=0
--tserver_master_addrs=127.13.251.190:34609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.133
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:03.349264 14945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:03.350970 14945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:08:03.363900 14945 server_base.cc:1034] running on GCE node
W20240501 20:08:03.364053 14951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:03.364501 14953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:03.364522 14950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:03.365334 14945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:03.366355 14945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:03.367599 14945 hybrid_clock.cc:648] HybridClock initialized: now 1714594083367505 us; error 57 us; skew 500 ppm
I20240501 20:08:03.367908 14945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:03.370374 14945 webserver.cc:460] Webserver started at http://127.13.251.133:42255/ using document root <none> and password file <none>
I20240501 20:08:03.371178 14945 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:03.371309 14945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:03.371791 14945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:03.374619 14945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e8bed3785648455fbbfd84c446260c00"
format_stamp: "Formatted at 2024-05-01 20:08:03 on dist-test-slave-qswb"
I20240501 20:08:03.375376 14945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e8bed3785648455fbbfd84c446260c00"
format_stamp: "Formatted at 2024-05-01 20:08:03 on dist-test-slave-qswb"
I20240501 20:08:03.380769 14945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:03.384255 14959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:03.385764 14945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:03.385926 14945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e8bed3785648455fbbfd84c446260c00"
format_stamp: "Formatted at 2024-05-01 20:08:03 on dist-test-slave-qswb"
I20240501 20:08:03.386160 14945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:03.403156 14945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:03.404019 14945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:03.404275 14945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:03.405198 14945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:03.406716 14945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:03.406788 14945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:03.406875 14945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:03.406925 14945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:03.438797 14945 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.133:34143
I20240501 20:08:03.438871 15072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.133:34143 every 8 connection(s)
I20240501 20:08:03.440204 14945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:03.446975 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 14945
I20240501 20:08:03.447170 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:03.453368 15073 heartbeater.cc:347] Connected to a master server at 127.13.251.190:34609
I20240501 20:08:03.453666 15073 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:03.454450 15073 heartbeater.cc:511] Master 127.13.251.190:34609 requested a full tablet report, sending...
I20240501 20:08:03.455761 14361 ts_manager.cc:194] Registered new tserver with Master: e8bed3785648455fbbfd84c446260c00 (127.13.251.133:34143)
I20240501 20:08:03.456476 14361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.133:57859
I20240501 20:08:03.464298 14318 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:03.484648 14361 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49926:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:08:03.526137 14482 tablet_service.cc:1467] Processing CreateTablet for tablet 3e26655ae6fb4775b55c030e128be223 (DEFAULT_TABLE table=TestTable [id=0a5dacea1a264122836bc273683da2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:03.527117 14745 tablet_service.cc:1467] Processing CreateTablet for tablet 3e26655ae6fb4775b55c030e128be223 (DEFAULT_TABLE table=TestTable [id=0a5dacea1a264122836bc273683da2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:03.527201 14876 tablet_service.cc:1467] Processing CreateTablet for tablet 3e26655ae6fb4775b55c030e128be223 (DEFAULT_TABLE table=TestTable [id=0a5dacea1a264122836bc273683da2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:03.527817 14482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e26655ae6fb4775b55c030e128be223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:03.528299 14745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e26655ae6fb4775b55c030e128be223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:03.528534 14876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e26655ae6fb4775b55c030e128be223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:03.538779 15091 tablet_bootstrap.cc:492] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb: Bootstrap starting.
I20240501 20:08:03.541687 15093 tablet_bootstrap.cc:492] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b: Bootstrap starting.
I20240501 20:08:03.542438 15092 tablet_bootstrap.cc:492] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace: Bootstrap starting.
I20240501 20:08:03.542753 15091 tablet_bootstrap.cc:654] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:03.544111 15091 log.cc:824] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:03.545264 15093 tablet_bootstrap.cc:654] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:03.545476 15092 tablet_bootstrap.cc:654] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:03.546703 15092 log.cc:824] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:03.546744 15093 log.cc:824] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:03.554827 15091 tablet_bootstrap.cc:492] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb: No bootstrap required, opened a new log
I20240501 20:08:03.555192 15091 ts_tablet_manager.cc:1367] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20240501 20:08:03.556754 15093 tablet_bootstrap.cc:492] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b: No bootstrap required, opened a new log
I20240501 20:08:03.556757 15092 tablet_bootstrap.cc:492] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace: No bootstrap required, opened a new log
I20240501 20:08:03.557044 15092 ts_tablet_manager.cc:1367] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20240501 20:08:03.557044 15093 ts_tablet_manager.cc:1367] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240501 20:08:03.563995 15091 raft_consensus.cc:348] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } }
I20240501 20:08:03.564570 15091 raft_consensus.cc:374] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:03.564725 15091 raft_consensus.cc:729] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7f96e0745646ec9f5bc9d3f3aeb8eb, State: Initialized, Role: FOLLOWER
I20240501 20:08:03.565557 15091 consensus_queue.cc:260] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } }
I20240501 20:08:03.566048 15092 raft_consensus.cc:348] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } }
I20240501 20:08:03.566473 15092 raft_consensus.cc:374] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:03.566581 15092 raft_consensus.cc:729] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b863e2a9eb8440d1a876b4e67e49aace, State: Initialized, Role: FOLLOWER
I20240501 20:08:03.566762 14810 heartbeater.cc:503] Master 127.13.251.190:34609 was elected leader, sending a full tablet report...
I20240501 20:08:03.566789 15093 raft_consensus.cc:348] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } }
I20240501 20:08:03.567207 15093 raft_consensus.cc:374] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:03.567307 15093 raft_consensus.cc:729] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd294148300c4c3a9fb856d7e2f4121b, State: Initialized, Role: FOLLOWER
I20240501 20:08:03.567365 15092 consensus_queue.cc:260] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } }
I20240501 20:08:03.568197 15093 consensus_queue.cc:260] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } }
I20240501 20:08:03.568470 15091 ts_tablet_manager.cc:1398] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20240501 20:08:03.574281 15093 ts_tablet_manager.cc:1398] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20240501 20:08:03.575358 14942 heartbeater.cc:503] Master 127.13.251.190:34609 was elected leader, sending a full tablet report...
I20240501 20:08:03.575910 15092 ts_tablet_manager.cc:1398] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
W20240501 20:08:03.579743 14548 tablet.cc:2369] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:03.732314 15097 raft_consensus.cc:482] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:03.732564 15097 raft_consensus.cc:504] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } }
I20240501 20:08:03.734328 15097 leader_election.cc:288] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd294148300c4c3a9fb856d7e2f4121b (127.13.251.129:34563), b863e2a9eb8440d1a876b4e67e49aace (127.13.251.132:36507)
W20240501 20:08:03.740965 14943 tablet.cc:2369] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:03.741601 14897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e26655ae6fb4775b55c030e128be223" candidate_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b863e2a9eb8440d1a876b4e67e49aace" is_pre_election: true
I20240501 20:08:03.741642 14502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e26655ae6fb4775b55c030e128be223" candidate_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd294148300c4c3a9fb856d7e2f4121b" is_pre_election: true
I20240501 20:08:03.742146 14502 raft_consensus.cc:2454] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e7f96e0745646ec9f5bc9d3f3aeb8eb in term 0.
I20240501 20:08:03.742148 14897 raft_consensus.cc:2454] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e7f96e0745646ec9f5bc9d3f3aeb8eb in term 0.
I20240501 20:08:03.742739 14700 leader_election.cc:302] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7f96e0745646ec9f5bc9d3f3aeb8eb, cd294148300c4c3a9fb856d7e2f4121b; no voters:
I20240501 20:08:03.743110 15097 raft_consensus.cc:2781] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:03.743261 15097 raft_consensus.cc:482] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:03.743363 15097 raft_consensus.cc:3037] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:03.745486 15097 raft_consensus.cc:504] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } }
I20240501 20:08:03.746075 15097 leader_election.cc:288] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [CANDIDATE]: Term 1 election: Requested vote from peers cd294148300c4c3a9fb856d7e2f4121b (127.13.251.129:34563), b863e2a9eb8440d1a876b4e67e49aace (127.13.251.132:36507)
I20240501 20:08:03.746594 14502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e26655ae6fb4775b55c030e128be223" candidate_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd294148300c4c3a9fb856d7e2f4121b"
I20240501 20:08:03.746654 14897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e26655ae6fb4775b55c030e128be223" candidate_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b863e2a9eb8440d1a876b4e67e49aace"
I20240501 20:08:03.746774 14502 raft_consensus.cc:3037] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:03.746851 14897 raft_consensus.cc:3037] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:03.748989 14502 raft_consensus.cc:2454] T 3e26655ae6fb4775b55c030e128be223 P cd294148300c4c3a9fb856d7e2f4121b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e7f96e0745646ec9f5bc9d3f3aeb8eb in term 1.
I20240501 20:08:03.748997 14897 raft_consensus.cc:2454] T 3e26655ae6fb4775b55c030e128be223 P b863e2a9eb8440d1a876b4e67e49aace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e7f96e0745646ec9f5bc9d3f3aeb8eb in term 1.
I20240501 20:08:03.749429 14700 leader_election.cc:302] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7f96e0745646ec9f5bc9d3f3aeb8eb, cd294148300c4c3a9fb856d7e2f4121b; no voters:
I20240501 20:08:03.749742 15097 raft_consensus.cc:2781] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:03.750152 15097 raft_consensus.cc:686] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [term 1 LEADER]: Becoming Leader. State: Replica: 0e7f96e0745646ec9f5bc9d3f3aeb8eb, State: Running, Role: LEADER
I20240501 20:08:03.750674 15097 consensus_queue.cc:237] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } }
I20240501 20:08:03.755885 14359 catalog_manager.cc:5484] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e7f96e0745646ec9f5bc9d3f3aeb8eb (127.13.251.131). New cstate: current_term: 1 leader_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e7f96e0745646ec9f5bc9d3f3aeb8eb" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 } health_report { overall_health: UNKNOWN } } }
W20240501 20:08:03.800374 14811 tablet.cc:2369] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:03.816313 14318 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:03.818753 14318 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cd294148300c4c3a9fb856d7e2f4121b to finish bootstrapping
I20240501 20:08:03.828008 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1bc51c3f33d141e79c9cc3db868cedfc to finish bootstrapping
I20240501 20:08:03.836750 14318 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0e7f96e0745646ec9f5bc9d3f3aeb8eb to finish bootstrapping
I20240501 20:08:03.844224 14318 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b863e2a9eb8440d1a876b4e67e49aace to finish bootstrapping
I20240501 20:08:03.851069 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e8bed3785648455fbbfd84c446260c00 to finish bootstrapping
I20240501 20:08:04.295217 15130 consensus_queue.cc:1035] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b863e2a9eb8440d1a876b4e67e49aace" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 36507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:04.315361 15133 consensus_queue.cc:1035] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd294148300c4c3a9fb856d7e2f4121b" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 20:08:04.332394 14745 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:04.344831 14613 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:04.367837 14876 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:04.417531 14482 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:04.432022 15008 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:04.458894 15073 heartbeater.cc:503] Master 127.13.251.190:34609 was elected leader, sending a full tablet report...
I20240501 20:08:04.578363 14613 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:04.592150 14876 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:04.596889 15008 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:04.603901 14745 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:04.605683 14482 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:04.755313 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 14420
I20240501 20:08:04.756858 14542 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:04.850821 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 14420
I20240501 20:08:04.875135 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 14551
I20240501 20:08:04.876204 14673 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:08:04.889876 14700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.129:34563: connect: Connection refused (error 111)
W20240501 20:08:04.891604 14700 consensus_peers.cc:485] T 3e26655ae6fb4775b55c030e128be223 P 0e7f96e0745646ec9f5bc9d3f3aeb8eb -> Peer cd294148300c4c3a9fb856d7e2f4121b (127.13.251.129:34563): Couldn't send request to peer cd294148300c4c3a9fb856d7e2f4121b. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.129:34563: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:04.960376 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 14551
I20240501 20:08:04.977771 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 14682
I20240501 20:08:04.978812 14804 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:05.069963 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 14682
I20240501 20:08:05.087692 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 14813
I20240501 20:08:05.088737 14937 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:05.179641 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 14813
I20240501 20:08:05.197037 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 14945
I20240501 20:08:05.198061 15067 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:05.282965 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 14945
I20240501 20:08:05.300034 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 14330
I20240501 20:08:05.301246 14391 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:05.408195 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 14330
2024-05-01T20:08:05Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (4895 ms)
[----------] 1 test from AdminCliTest (4895 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240501 20:08:05.451752 14318 test_util.cc:268] Using random seed: 191215548
I20240501 20:08:05.453641 14318 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:05.453737 14318 ts_itest-base.cc:113] --------------
I20240501 20:08:05.453797 14318 ts_itest-base.cc:114] 5 tablet servers
I20240501 20:08:05.453872 14318 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:08:05.453925 14318 ts_itest-base.cc:116] --------------
2024-05-01T20:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:05Z Disabled control of system clock
I20240501 20:08:05.477708 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.251.190:42903
--webserver_interface=127.13.251.190
--webserver_port=0
--builtin_ntp_servers=127.13.251.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.251.190:42903
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 20:08:05.824698 15204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:05.825059 15204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:05.825201 15204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:05.833081 15204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:05.833250 15204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:05.833328 15204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:05.833415 15204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:05.833482 15204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:05.843528 15204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.13.251.190:42903
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.251.190:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.251.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:05.845122 15204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:05.847235 15204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:05.859997 15212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:05.860738 15210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:05.860778 15209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.861364 15204 server_base.cc:1034] running on GCE node
I20240501 20:08:05.862222 15204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:05.863929 15204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:05.865164 15204 hybrid_clock.cc:648] HybridClock initialized: now 1714594085865100 us; error 46 us; skew 500 ppm
I20240501 20:08:05.865573 15204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:05.868167 15204 webserver.cc:460] Webserver started at http://127.13.251.190:44429/ using document root <none> and password file <none>
I20240501 20:08:05.868957 15204 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:05.869093 15204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:05.869428 15204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:05.872198 15204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "af892ecc046d4c36bd14ab00b9747c07"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-qswb"
I20240501 20:08:05.873031 15204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "af892ecc046d4c36bd14ab00b9747c07"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-qswb"
I20240501 20:08:05.879244 15204 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240501 20:08:05.883055 15218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.884678 15204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:05.884902 15204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
uuid: "af892ecc046d4c36bd14ab00b9747c07"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-qswb"
I20240501 20:08:05.885118 15204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:05.905470 15204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:05.906419 15204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:05.906728 15204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:05.926451 15204 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.190:42903
I20240501 20:08:05.926484 15270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.190:42903 every 8 connection(s)
I20240501 20:08:05.927973 15204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:05.931744 15271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:05.935889 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 15204
I20240501 20:08:05.936112 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:05.942103 15271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07: Bootstrap starting.
I20240501 20:08:05.945384 15271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:05.946483 15271 log.cc:824] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:05.949425 15271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07: No bootstrap required, opened a new log
I20240501 20:08:05.956403 15271 raft_consensus.cc:348] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af892ecc046d4c36bd14ab00b9747c07" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 42903 } }
I20240501 20:08:05.956760 15271 raft_consensus.cc:374] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:05.956881 15271 raft_consensus.cc:729] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af892ecc046d4c36bd14ab00b9747c07, State: Initialized, Role: FOLLOWER
I20240501 20:08:05.957527 15271 consensus_queue.cc:260] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af892ecc046d4c36bd14ab00b9747c07" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 42903 } }
I20240501 20:08:05.957728 15271 raft_consensus.cc:388] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:05.957871 15271 raft_consensus.cc:482] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:05.958034 15271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:05.960191 15271 raft_consensus.cc:504] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af892ecc046d4c36bd14ab00b9747c07" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 42903 } }
I20240501 20:08:05.960669 15271 leader_election.cc:302] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af892ecc046d4c36bd14ab00b9747c07; no voters:
I20240501 20:08:05.961285 15271 leader_election.cc:288] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:05.961400 15276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:05.962081 15276 raft_consensus.cc:686] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [term 1 LEADER]: Becoming Leader. State: Replica: af892ecc046d4c36bd14ab00b9747c07, State: Running, Role: LEADER
I20240501 20:08:05.962668 15276 consensus_queue.cc:237] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af892ecc046d4c36bd14ab00b9747c07" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 42903 } }
I20240501 20:08:05.963565 15271 sys_catalog.cc:564] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:05.965884 15277 sys_catalog.cc:455] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af892ecc046d4c36bd14ab00b9747c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af892ecc046d4c36bd14ab00b9747c07" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 42903 } } }
I20240501 20:08:05.966313 15277 sys_catalog.cc:458] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:05.967295 15278 sys_catalog.cc:455] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af892ecc046d4c36bd14ab00b9747c07. Latest consensus state: current_term: 1 leader_uuid: "af892ecc046d4c36bd14ab00b9747c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af892ecc046d4c36bd14ab00b9747c07" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 42903 } } }
I20240501 20:08:05.968274 15278 sys_catalog.cc:458] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:05.968343 15284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:05.974720 15284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:05.984745 15284 catalog_manager.cc:1347] Generated new cluster ID: 5744393eb50e49f49e1f32aea844ea92
I20240501 20:08:05.984897 15284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:06.003998 15284 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:06.005941 15284 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:06.028885 15284 catalog_manager.cc:5857] T 00000000000000000000000000000000 P af892ecc046d4c36bd14ab00b9747c07: Generated new TSK 0
I20240501 20:08:06.030149 15284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:06.048372 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.129:0
--local_ip_for_outbound_sockets=127.13.251.129
--webserver_interface=127.13.251.129
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--builtin_ntp_servers=127.13.251.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:08:06.416240 15295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:06.416564 15295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:06.416710 15295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:06.424928 15295 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:06.425108 15295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:06.425287 15295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.129
I20240501 20:08:06.435171 15295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.251.129
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.129
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:06.436676 15295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:06.438838 15295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:06.452231 15301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:06.452442 15303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:06.452980 15300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:06.453652 15295 server_base.cc:1034] running on GCE node
I20240501 20:08:06.454337 15295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:06.455490 15295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:06.456740 15295 hybrid_clock.cc:648] HybridClock initialized: now 1714594086456681 us; error 54 us; skew 500 ppm
I20240501 20:08:06.457104 15295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:06.459607 15295 webserver.cc:460] Webserver started at http://127.13.251.129:38257/ using document root <none> and password file <none>
I20240501 20:08:06.460397 15295 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:06.460518 15295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:06.460848 15295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:06.463583 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "524807a7875d428d9e5c46002fd60241"
format_stamp: "Formatted at 2024-05-01 20:08:06 on dist-test-slave-qswb"
I20240501 20:08:06.464388 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "524807a7875d428d9e5c46002fd60241"
format_stamp: "Formatted at 2024-05-01 20:08:06 on dist-test-slave-qswb"
I20240501 20:08:06.469944 15295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:06.473601 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:06.475045 15295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:06.475236 15295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "524807a7875d428d9e5c46002fd60241"
format_stamp: "Formatted at 2024-05-01 20:08:06 on dist-test-slave-qswb"
I20240501 20:08:06.475455 15295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:06.494294 15295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:06.495358 15295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:06.495718 15295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:06.496757 15295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:06.498343 15295 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:06.498442 15295 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:06.498553 15295 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:06.498623 15295 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:06.532450 15295 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.129:44531
I20240501 20:08:06.532498 15422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.129:44531 every 8 connection(s)
I20240501 20:08:06.533790 15295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:06.539386 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 15295
I20240501 20:08:06.539613 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:06.543408 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.130:0
--local_ip_for_outbound_sockets=127.13.251.130
--webserver_interface=127.13.251.130
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--builtin_ntp_servers=127.13.251.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:06.553632 15423 heartbeater.cc:347] Connected to a master server at 127.13.251.190:42903
I20240501 20:08:06.553980 15423 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:06.554792 15423 heartbeater.cc:511] Master 127.13.251.190:42903 requested a full tablet report, sending...
I20240501 20:08:06.556943 15236 ts_manager.cc:194] Registered new tserver with Master: 524807a7875d428d9e5c46002fd60241 (127.13.251.129:44531)
I20240501 20:08:06.558477 15236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.129:47475
W20240501 20:08:06.887854 15427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:06.888186 15427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:06.888304 15427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:06.896255 15427 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:06.896433 15427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:06.896605 15427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.130
I20240501 20:08:06.906001 15427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.251.130
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.130
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:06.907730 15427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:06.909551 15427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:06.922207 15433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:06.922760 15435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:06.923013 15432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:06.923724 15427 server_base.cc:1034] running on GCE node
I20240501 20:08:06.924366 15427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:06.925386 15427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:06.926607 15427 hybrid_clock.cc:648] HybridClock initialized: now 1714594086926550 us; error 53 us; skew 500 ppm
I20240501 20:08:06.927031 15427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:06.929867 15427 webserver.cc:460] Webserver started at http://127.13.251.130:41453/ using document root <none> and password file <none>
I20240501 20:08:06.930845 15427 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:06.930968 15427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:06.931340 15427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:06.935045 15427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b0dd2345af404f1cac59d18c3ace658f"
format_stamp: "Formatted at 2024-05-01 20:08:06 on dist-test-slave-qswb"
I20240501 20:08:06.936112 15427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b0dd2345af404f1cac59d18c3ace658f"
format_stamp: "Formatted at 2024-05-01 20:08:06 on dist-test-slave-qswb"
I20240501 20:08:06.943488 15427 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240501 20:08:06.947983 15441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:06.949417 15427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:06.949580 15427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b0dd2345af404f1cac59d18c3ace658f"
format_stamp: "Formatted at 2024-05-01 20:08:06 on dist-test-slave-qswb"
I20240501 20:08:06.949785 15427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:06.969339 15427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:06.970201 15427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:06.970461 15427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:06.971415 15427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:06.972908 15427 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:06.973001 15427 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:06.973109 15427 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:06.973182 15427 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:07.005722 15427 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.130:44875
I20240501 20:08:07.005781 15554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.130:44875 every 8 connection(s)
I20240501 20:08:07.007129 15427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:07.012615 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 15427
I20240501 20:08:07.012836 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:07.016213 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.131:0
--local_ip_for_outbound_sockets=127.13.251.131
--webserver_interface=127.13.251.131
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--builtin_ntp_servers=127.13.251.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:07.022223 15555 heartbeater.cc:347] Connected to a master server at 127.13.251.190:42903
I20240501 20:08:07.022552 15555 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:07.023311 15555 heartbeater.cc:511] Master 127.13.251.190:42903 requested a full tablet report, sending...
I20240501 20:08:07.024794 15236 ts_manager.cc:194] Registered new tserver with Master: b0dd2345af404f1cac59d18c3ace658f (127.13.251.130:44875)
I20240501 20:08:07.025913 15236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.130:55803
W20240501 20:08:07.362198 15558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:07.362529 15558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:07.362620 15558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:07.370538 15558 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:07.370680 15558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:07.370797 15558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.131
I20240501 20:08:07.380321 15558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.251.131
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.131
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:07.381764 15558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:07.383493 15558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:07.396059 15566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:07.396605 15563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:07.396879 15564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:07.397411 15558 server_base.cc:1034] running on GCE node
I20240501 20:08:07.398061 15558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:07.399188 15558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:07.400401 15558 hybrid_clock.cc:648] HybridClock initialized: now 1714594087400350 us; error 48 us; skew 500 ppm
I20240501 20:08:07.400736 15558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:07.403185 15558 webserver.cc:460] Webserver started at http://127.13.251.131:43995/ using document root <none> and password file <none>
I20240501 20:08:07.404008 15558 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:07.404134 15558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:07.404460 15558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:07.407254 15558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c30380fda50b41d8bad38ef4456e265c"
format_stamp: "Formatted at 2024-05-01 20:08:07 on dist-test-slave-qswb"
I20240501 20:08:07.408071 15558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c30380fda50b41d8bad38ef4456e265c"
format_stamp: "Formatted at 2024-05-01 20:08:07 on dist-test-slave-qswb"
I20240501 20:08:07.413470 15558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20240501 20:08:07.417145 15572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:07.418550 15558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:07.418720 15558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c30380fda50b41d8bad38ef4456e265c"
format_stamp: "Formatted at 2024-05-01 20:08:07 on dist-test-slave-qswb"
I20240501 20:08:07.418941 15558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:07.434809 15558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:07.435701 15558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:07.435998 15558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:07.436933 15558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:07.438373 15558 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:07.438462 15558 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:07.438547 15558 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:07.438601 15558 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:07.471096 15558 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.131:41245
I20240501 20:08:07.471143 15685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.131:41245 every 8 connection(s)
I20240501 20:08:07.472440 15558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:07.477075 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 15558
I20240501 20:08:07.477211 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:07.480556 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.132:0
--local_ip_for_outbound_sockets=127.13.251.132
--webserver_interface=127.13.251.132
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--builtin_ntp_servers=127.13.251.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:07.486455 15686 heartbeater.cc:347] Connected to a master server at 127.13.251.190:42903
I20240501 20:08:07.486759 15686 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:07.487516 15686 heartbeater.cc:511] Master 127.13.251.190:42903 requested a full tablet report, sending...
I20240501 20:08:07.489101 15236 ts_manager.cc:194] Registered new tserver with Master: c30380fda50b41d8bad38ef4456e265c (127.13.251.131:41245)
I20240501 20:08:07.490321 15236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.131:36917
I20240501 20:08:07.561424 15423 heartbeater.cc:503] Master 127.13.251.190:42903 was elected leader, sending a full tablet report...
W20240501 20:08:07.815435 15689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:07.815788 15689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:07.815891 15689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:07.823457 15689 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:07.823635 15689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:07.823791 15689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.132
I20240501 20:08:07.833216 15689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.13.251.132
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.132
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:07.834591 15689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:07.836385 15689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:07.848299 15695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:07.848793 15694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:07.849519 15697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:07.850209 15689 server_base.cc:1034] running on GCE node
I20240501 20:08:07.850811 15689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:07.851764 15689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:07.852969 15689 hybrid_clock.cc:648] HybridClock initialized: now 1714594087852917 us; error 51 us; skew 500 ppm
I20240501 20:08:07.853322 15689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:07.855604 15689 webserver.cc:460] Webserver started at http://127.13.251.132:39511/ using document root <none> and password file <none>
I20240501 20:08:07.856328 15689 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:07.856437 15689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:07.856752 15689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:07.859275 15689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "79bcd51327c24dbcbf3c77e5eccf706d"
format_stamp: "Formatted at 2024-05-01 20:08:07 on dist-test-slave-qswb"
I20240501 20:08:07.860008 15689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "79bcd51327c24dbcbf3c77e5eccf706d"
format_stamp: "Formatted at 2024-05-01 20:08:07 on dist-test-slave-qswb"
I20240501 20:08:07.864910 15689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:07.868116 15703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:07.869407 15689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:07.869560 15689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "79bcd51327c24dbcbf3c77e5eccf706d"
format_stamp: "Formatted at 2024-05-01 20:08:07 on dist-test-slave-qswb"
I20240501 20:08:07.869742 15689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:07.883335 15689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:07.884209 15689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:07.884475 15689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:07.885404 15689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:07.886842 15689 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:07.886946 15689 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:07.887060 15689 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:07.887131 15689 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:07.919713 15689 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.132:38749
I20240501 20:08:07.919767 15815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.132:38749 every 8 connection(s)
I20240501 20:08:07.921002 15689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:07.924589 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 15689
I20240501 20:08:07.924780 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:07.928262 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.133:0
--local_ip_for_outbound_sockets=127.13.251.133
--webserver_interface=127.13.251.133
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--builtin_ntp_servers=127.13.251.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:07.934487 15816 heartbeater.cc:347] Connected to a master server at 127.13.251.190:42903
I20240501 20:08:07.934791 15816 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:07.935511 15816 heartbeater.cc:511] Master 127.13.251.190:42903 requested a full tablet report, sending...
I20240501 20:08:07.936828 15236 ts_manager.cc:194] Registered new tserver with Master: 79bcd51327c24dbcbf3c77e5eccf706d (127.13.251.132:38749)
I20240501 20:08:07.937623 15236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.132:44377
I20240501 20:08:08.028774 15555 heartbeater.cc:503] Master 127.13.251.190:42903 was elected leader, sending a full tablet report...
W20240501 20:08:08.263051 15819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:08.263381 15819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:08.263522 15819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:08.271055 15819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:08.271203 15819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:08.271370 15819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.133
I20240501 20:08:08.280560 15819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.13.251.133
--webserver_port=0
--tserver_master_addrs=127.13.251.190:42903
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.133
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:08.281986 15819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:08.283771 15819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:08.296119 15824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:08.296643 15825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:08.297214 15827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:08.298087 15819 server_base.cc:1034] running on GCE node
I20240501 20:08:08.298702 15819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:08.299731 15819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:08.300974 15819 hybrid_clock.cc:648] HybridClock initialized: now 1714594088300929 us; error 37 us; skew 500 ppm
I20240501 20:08:08.301323 15819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:08.303607 15819 webserver.cc:460] Webserver started at http://127.13.251.133:41763/ using document root <none> and password file <none>
I20240501 20:08:08.304351 15819 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:08.304466 15819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:08.304776 15819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:08.307350 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "dc777a7b1574484fa200d4138253b367"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-qswb"
I20240501 20:08:08.308125 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "dc777a7b1574484fa200d4138253b367"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-qswb"
I20240501 20:08:08.313428 15819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:08.316910 15833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.318300 15819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:08.318449 15819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "dc777a7b1574484fa200d4138253b367"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-qswb"
I20240501 20:08:08.318639 15819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:08.352224 15819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:08.353116 15819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:08.353376 15819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:08.354357 15819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:08.355938 15819 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:08.356025 15819 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.356137 15819 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:08.356199 15819 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.386756 15819 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.133:43113
I20240501 20:08:08.386813 15945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.133:43113 every 8 connection(s)
I20240501 20:08:08.388159 15819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:08.394516 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 15819
I20240501 20:08:08.394691 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:08.401636 15946 heartbeater.cc:347] Connected to a master server at 127.13.251.190:42903
I20240501 20:08:08.401937 15946 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:08.402628 15946 heartbeater.cc:511] Master 127.13.251.190:42903 requested a full tablet report, sending...
I20240501 20:08:08.403892 15236 ts_manager.cc:194] Registered new tserver with Master: dc777a7b1574484fa200d4138253b367 (127.13.251.133:43113)
I20240501 20:08:08.404628 15236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.133:57671
I20240501 20:08:08.410275 14318 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:08.425527 15236 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55774:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:08:08.463112 15751 tablet_service.cc:1467] Processing CreateTablet for tablet 10896c7ca12f4cd59fc91159b8ccc2af (DEFAULT_TABLE table=TestTable [id=ec89e8b89ed54de2a5caefab012e825e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:08.464918 15751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10896c7ca12f4cd59fc91159b8ccc2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:08.470217 15358 tablet_service.cc:1467] Processing CreateTablet for tablet 10896c7ca12f4cd59fc91159b8ccc2af (DEFAULT_TABLE table=TestTable [id=ec89e8b89ed54de2a5caefab012e825e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:08.471005 15621 tablet_service.cc:1467] Processing CreateTablet for tablet 10896c7ca12f4cd59fc91159b8ccc2af (DEFAULT_TABLE table=TestTable [id=ec89e8b89ed54de2a5caefab012e825e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:08.471940 15358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10896c7ca12f4cd59fc91159b8ccc2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:08.472447 15621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10896c7ca12f4cd59fc91159b8ccc2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:08.485142 15964 tablet_bootstrap.cc:492] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d: Bootstrap starting.
I20240501 20:08:08.488173 15965 tablet_bootstrap.cc:492] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c: Bootstrap starting.
I20240501 20:08:08.489373 15964 tablet_bootstrap.cc:654] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:08.489434 15966 tablet_bootstrap.cc:492] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241: Bootstrap starting.
I20240501 20:08:08.490593 15964 log.cc:824] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:08.492487 15965 tablet_bootstrap.cc:654] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:08.492949 15686 heartbeater.cc:503] Master 127.13.251.190:42903 was elected leader, sending a full tablet report...
I20240501 20:08:08.493598 15964 tablet_bootstrap.cc:492] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d: No bootstrap required, opened a new log
I20240501 20:08:08.493651 15965 log.cc:824] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:08.493893 15964 ts_tablet_manager.cc:1367] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:08:08.494781 15966 tablet_bootstrap.cc:654] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:08.495985 15966 log.cc:824] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:08.502097 15965 tablet_bootstrap.cc:492] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c: No bootstrap required, opened a new log
I20240501 20:08:08.502350 15965 ts_tablet_manager.cc:1367] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240501 20:08:08.503062 15964 raft_consensus.cc:348] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:08.503393 15964 raft_consensus.cc:374] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:08.503490 15964 raft_consensus.cc:729] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bcd51327c24dbcbf3c77e5eccf706d, State: Initialized, Role: FOLLOWER
I20240501 20:08:08.504218 15964 consensus_queue.cc:260] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:08.505316 15816 heartbeater.cc:503] Master 127.13.251.190:42903 was elected leader, sending a full tablet report...
I20240501 20:08:08.505630 15966 tablet_bootstrap.cc:492] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241: No bootstrap required, opened a new log
I20240501 20:08:08.505908 15964 ts_tablet_manager.cc:1398] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20240501 20:08:08.505898 15966 ts_tablet_manager.cc:1367] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20240501 20:08:08.509555 15965 raft_consensus.cc:348] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:08.509989 15965 raft_consensus.cc:374] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:08.510085 15965 raft_consensus.cc:729] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30380fda50b41d8bad38ef4456e265c, State: Initialized, Role: FOLLOWER
I20240501 20:08:08.510722 15965 consensus_queue.cc:260] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:08.512565 15965 ts_tablet_manager.cc:1398] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s
I20240501 20:08:08.513150 15966 raft_consensus.cc:348] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:08.513566 15966 raft_consensus.cc:374] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:08.513686 15966 raft_consensus.cc:729] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524807a7875d428d9e5c46002fd60241, State: Initialized, Role: FOLLOWER
I20240501 20:08:08.514364 15966 consensus_queue.cc:260] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:08.515972 15966 ts_tablet_manager.cc:1398] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
W20240501 20:08:08.537065 15424 tablet.cc:2369] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:08.636003 15972 raft_consensus.cc:482] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:08.636304 15972 raft_consensus.cc:504] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:08.637961 15972 leader_election.cc:288] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79bcd51327c24dbcbf3c77e5eccf706d (127.13.251.132:38749), c30380fda50b41d8bad38ef4456e265c (127.13.251.131:41245)
I20240501 20:08:08.646364 15771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10896c7ca12f4cd59fc91159b8ccc2af" candidate_uuid: "524807a7875d428d9e5c46002fd60241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" is_pre_election: true
I20240501 20:08:08.646520 15641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10896c7ca12f4cd59fc91159b8ccc2af" candidate_uuid: "524807a7875d428d9e5c46002fd60241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30380fda50b41d8bad38ef4456e265c" is_pre_election: true
I20240501 20:08:08.647027 15771 raft_consensus.cc:2454] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524807a7875d428d9e5c46002fd60241 in term 0.
I20240501 20:08:08.647028 15641 raft_consensus.cc:2454] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524807a7875d428d9e5c46002fd60241 in term 0.
I20240501 20:08:08.647750 15312 leader_election.cc:302] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524807a7875d428d9e5c46002fd60241, 79bcd51327c24dbcbf3c77e5eccf706d; no voters:
I20240501 20:08:08.648113 15972 raft_consensus.cc:2781] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:08.648263 15972 raft_consensus.cc:482] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:08.648356 15972 raft_consensus.cc:3037] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:08.650372 15972 raft_consensus.cc:504] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:08.650961 15972 leader_election.cc:288] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [CANDIDATE]: Term 1 election: Requested vote from peers 79bcd51327c24dbcbf3c77e5eccf706d (127.13.251.132:38749), c30380fda50b41d8bad38ef4456e265c (127.13.251.131:41245)
I20240501 20:08:08.651472 15771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10896c7ca12f4cd59fc91159b8ccc2af" candidate_uuid: "524807a7875d428d9e5c46002fd60241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79bcd51327c24dbcbf3c77e5eccf706d"
I20240501 20:08:08.651575 15641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10896c7ca12f4cd59fc91159b8ccc2af" candidate_uuid: "524807a7875d428d9e5c46002fd60241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30380fda50b41d8bad38ef4456e265c"
I20240501 20:08:08.651708 15771 raft_consensus.cc:3037] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:08.651789 15641 raft_consensus.cc:3037] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:08.653939 15641 raft_consensus.cc:2454] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524807a7875d428d9e5c46002fd60241 in term 1.
I20240501 20:08:08.653937 15771 raft_consensus.cc:2454] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524807a7875d428d9e5c46002fd60241 in term 1.
I20240501 20:08:08.654587 15312 leader_election.cc:302] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524807a7875d428d9e5c46002fd60241, c30380fda50b41d8bad38ef4456e265c; no voters:
I20240501 20:08:08.654907 15972 raft_consensus.cc:2781] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:08.655370 15972 raft_consensus.cc:686] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [term 1 LEADER]: Becoming Leader. State: Replica: 524807a7875d428d9e5c46002fd60241, State: Running, Role: LEADER
I20240501 20:08:08.655998 15972 consensus_queue.cc:237] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:08.660331 15234 catalog_manager.cc:5484] T 10896c7ca12f4cd59fc91159b8ccc2af P 524807a7875d428d9e5c46002fd60241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524807a7875d428d9e5c46002fd60241 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "524807a7875d428d9e5c46002fd60241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } } }
W20240501 20:08:08.672967 15817 tablet.cc:2369] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:08.691351 14318 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:08.693646 14318 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 524807a7875d428d9e5c46002fd60241 to finish bootstrapping
I20240501 20:08:08.702255 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b0dd2345af404f1cac59d18c3ace658f to finish bootstrapping
I20240501 20:08:08.711284 14318 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c30380fda50b41d8bad38ef4456e265c to finish bootstrapping
I20240501 20:08:08.718508 14318 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 79bcd51327c24dbcbf3c77e5eccf706d to finish bootstrapping
W20240501 20:08:08.724717 15687 tablet.cc:2369] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:08.726013 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dc777a7b1574484fa200d4138253b367 to finish bootstrapping
I20240501 20:08:08.734333 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 15295
I20240501 20:08:08.735272 15418 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:08.823647 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 15295
I20240501 20:08:09.294754 15751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.296497 15489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.296870 15621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.313656 15881 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.406854 15946 heartbeater.cc:503] Master 127.13.251.190:42903 was elected leader, sending a full tablet report...
I20240501 20:08:09.440261 15751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.445488 15621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.445811 15881 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.452982 15489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.617957 15751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.619172 15489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.620131 15621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:09.621779 15881 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.223488 15489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.224828 15751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.232642 15621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.232800 15881 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.284283 16115 raft_consensus.cc:482] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:10.284600 16115 raft_consensus.cc:504] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:10.286832 16115 leader_election.cc:288] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c30380fda50b41d8bad38ef4456e265c (127.13.251.131:41245), 524807a7875d428d9e5c46002fd60241 (127.13.251.129:44531)
W20240501 20:08:10.288570 15706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.129:44531: connect: Connection refused (error 111)
W20240501 20:08:10.293083 15706 leader_election.cc:334] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 524807a7875d428d9e5c46002fd60241 (127.13.251.129:44531): Network error: Client connection negotiation failed: client connection to 127.13.251.129:44531: connect: Connection refused (error 111)
I20240501 20:08:10.294521 15641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10896c7ca12f4cd59fc91159b8ccc2af" candidate_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30380fda50b41d8bad38ef4456e265c" is_pre_election: true
I20240501 20:08:10.294746 15641 raft_consensus.cc:2454] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79bcd51327c24dbcbf3c77e5eccf706d in term 1.
I20240501 20:08:10.295210 15706 leader_election.cc:302] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79bcd51327c24dbcbf3c77e5eccf706d, c30380fda50b41d8bad38ef4456e265c; no voters: 524807a7875d428d9e5c46002fd60241
I20240501 20:08:10.295605 16115 raft_consensus.cc:2781] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:08:10.295740 16115 raft_consensus.cc:482] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:10.295847 16115 raft_consensus.cc:3037] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:10.297776 16115 raft_consensus.cc:504] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:10.298400 16115 leader_election.cc:288] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [CANDIDATE]: Term 2 election: Requested vote from peers c30380fda50b41d8bad38ef4456e265c (127.13.251.131:41245), 524807a7875d428d9e5c46002fd60241 (127.13.251.129:44531)
I20240501 20:08:10.298873 15641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10896c7ca12f4cd59fc91159b8ccc2af" candidate_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30380fda50b41d8bad38ef4456e265c"
I20240501 20:08:10.299082 15641 raft_consensus.cc:3037] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:08:10.299858 15706 leader_election.cc:334] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 524807a7875d428d9e5c46002fd60241 (127.13.251.129:44531): Network error: Client connection negotiation failed: client connection to 127.13.251.129:44531: connect: Connection refused (error 111)
I20240501 20:08:10.301224 15641 raft_consensus.cc:2454] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79bcd51327c24dbcbf3c77e5eccf706d in term 2.
I20240501 20:08:10.301590 15706 leader_election.cc:302] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79bcd51327c24dbcbf3c77e5eccf706d, c30380fda50b41d8bad38ef4456e265c; no voters: 524807a7875d428d9e5c46002fd60241
I20240501 20:08:10.301869 16115 raft_consensus.cc:2781] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:10.302307 16115 raft_consensus.cc:686] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [term 2 LEADER]: Becoming Leader. State: Replica: 79bcd51327c24dbcbf3c77e5eccf706d, State: Running, Role: LEADER
I20240501 20:08:10.302889 16115 consensus_queue.cc:237] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } }
I20240501 20:08:10.308354 15234 catalog_manager.cc:5484] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d reported cstate change: term changed from 1 to 2, leader changed from 524807a7875d428d9e5c46002fd60241 (127.13.251.129) to 79bcd51327c24dbcbf3c77e5eccf706d (127.13.251.132). New cstate: current_term: 2 leader_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bcd51327c24dbcbf3c77e5eccf706d" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 38749 } } peers { permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 } } peers { permanent_uuid: "524807a7875d428d9e5c46002fd60241" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44531 } } }
I20240501 20:08:10.368785 15751 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:10.372308 15489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.373042 15621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.374704 15881 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.552115 15621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.552481 15489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.563117 15751 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:10.564033 15881 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:10.704818 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 15427
I20240501 20:08:10.706461 15549 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:10.799304 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 15427
W20240501 20:08:10.804553 15706 consensus_peers.cc:485] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d -> Peer 524807a7875d428d9e5c46002fd60241 (127.13.251.129:44531): Couldn't send request to peer 524807a7875d428d9e5c46002fd60241. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.129:44531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:10.810604 15641 raft_consensus.cc:1261] T 10896c7ca12f4cd59fc91159b8ccc2af P c30380fda50b41d8bad38ef4456e265c [term 2 FOLLOWER]: Refusing update from remote peer 79bcd51327c24dbcbf3c77e5eccf706d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 20:08:10.811707 16115 consensus_queue.cc:1035] T 10896c7ca12f4cd59fc91159b8ccc2af P 79bcd51327c24dbcbf3c77e5eccf706d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30380fda50b41d8bad38ef4456e265c" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 41245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.820631 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 15558
I20240501 20:08:10.821578 15681 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:10.920526 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 15558
I20240501 20:08:10.945789 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 15689
I20240501 20:08:10.946803 15810 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:11.045487 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 15689
I20240501 20:08:11.065346 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 15819
I20240501 20:08:11.066371 15941 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:11.158985 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 15819
I20240501 20:08:11.176925 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 15204
I20240501 20:08:11.178086 15266 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:11.293691 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 15204
2024-05-01T20:08:11Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (5885 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5885 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240501 20:08:11.337010 14318 test_util.cc:268] Using random seed: 197100827
I20240501 20:08:11.338876 14318 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:11.338941 14318 ts_itest-base.cc:113] --------------
I20240501 20:08:11.338996 14318 ts_itest-base.cc:114] 7 tablet servers
I20240501 20:08:11.339048 14318 ts_itest-base.cc:115] 1 replicas per TS
I20240501 20:08:11.339123 14318 ts_itest-base.cc:116] --------------
2024-05-01T20:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:11Z Disabled control of system clock
I20240501 20:08:11.362917 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.251.190:43847
--webserver_interface=127.13.251.190
--webserver_port=0
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.251.190:43847
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 20:08:11.688720 16198 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:08:11.689080 16198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:11.689196 16198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:11.689302 16198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:11.696741 16198 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:11.696887 16198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:11.696974 16198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:11.697038 16198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:11.697135 16198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:11.706208 16198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.13.251.190:43847
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.251.190:43847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.251.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:11.707603 16198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:11.709290 16198 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:11.719805 16203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:11.720522 16206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:11.721196 16204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:11.721462 16198 server_base.cc:1034] running on GCE node
I20240501 20:08:11.722182 16198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:11.723590 16198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:11.724808 16198 hybrid_clock.cc:648] HybridClock initialized: now 1714594091724749 us; error 32 us; skew 500 ppm
I20240501 20:08:11.725128 16198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:11.734423 16198 webserver.cc:460] Webserver started at http://127.13.251.190:45441/ using document root <none> and password file <none>
I20240501 20:08:11.735205 16198 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:11.735332 16198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:11.735697 16198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:11.738723 16198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "919cd28c68d8439dbca079e3b938e719"
format_stamp: "Formatted at 2024-05-01 20:08:11 on dist-test-slave-qswb"
I20240501 20:08:11.739497 16198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "919cd28c68d8439dbca079e3b938e719"
format_stamp: "Formatted at 2024-05-01 20:08:11 on dist-test-slave-qswb"
I20240501 20:08:11.745008 16198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240501 20:08:11.748389 16212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:11.749671 16198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:11.749835 16198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
uuid: "919cd28c68d8439dbca079e3b938e719"
format_stamp: "Formatted at 2024-05-01 20:08:11 on dist-test-slave-qswb"
I20240501 20:08:11.750020 16198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:11.769066 16198 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:11.769904 16198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:11.770203 16198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:11.788056 16198 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.190:43847
I20240501 20:08:11.788077 16263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.190:43847 every 8 connection(s)
I20240501 20:08:11.789479 16198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:11.790527 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 16198
I20240501 20:08:11.790735 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:11.794132 16264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:11.806164 16264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719: Bootstrap starting.
I20240501 20:08:11.809134 16264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:11.810114 16264 log.cc:824] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:11.812752 16264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719: No bootstrap required, opened a new log
I20240501 20:08:11.818794 16264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919cd28c68d8439dbca079e3b938e719" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 43847 } }
I20240501 20:08:11.819139 16264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:11.819247 16264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919cd28c68d8439dbca079e3b938e719, State: Initialized, Role: FOLLOWER
I20240501 20:08:11.819919 16264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919cd28c68d8439dbca079e3b938e719" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 43847 } }
I20240501 20:08:11.820116 16264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:11.820237 16264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:11.820376 16264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:11.822430 16264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919cd28c68d8439dbca079e3b938e719" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 43847 } }
I20240501 20:08:11.822968 16264 leader_election.cc:302] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 919cd28c68d8439dbca079e3b938e719; no voters:
I20240501 20:08:11.823626 16264 leader_election.cc:288] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:11.823772 16269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:11.824396 16269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [term 1 LEADER]: Becoming Leader. State: Replica: 919cd28c68d8439dbca079e3b938e719, State: Running, Role: LEADER
I20240501 20:08:11.824950 16269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919cd28c68d8439dbca079e3b938e719" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 43847 } }
I20240501 20:08:11.825542 16264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:11.827602 16270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "919cd28c68d8439dbca079e3b938e719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919cd28c68d8439dbca079e3b938e719" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 43847 } } }
I20240501 20:08:11.827971 16270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:11.827664 16271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 919cd28c68d8439dbca079e3b938e719. Latest consensus state: current_term: 1 leader_uuid: "919cd28c68d8439dbca079e3b938e719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919cd28c68d8439dbca079e3b938e719" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 43847 } } }
I20240501 20:08:11.828369 16271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:11.830085 16277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:11.834929 16277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:11.843194 16277 catalog_manager.cc:1347] Generated new cluster ID: 3e28bfaaa7db4f73809f636c4b7300b4
I20240501 20:08:11.843309 16277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:11.876166 16277 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:11.877892 16277 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:11.891645 16277 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 919cd28c68d8439dbca079e3b938e719: Generated new TSK 0
I20240501 20:08:11.892760 16277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:11.901849 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.129:0
--local_ip_for_outbound_sockets=127.13.251.129
--webserver_interface=127.13.251.129
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:08:12.239851 16288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:12.240188 16288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:12.240324 16288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:12.248026 16288 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:12.248164 16288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:12.248320 16288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.129
I20240501 20:08:12.257881 16288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.251.129
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.129
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:12.259276 16288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:12.261058 16288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:12.273276 16293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:12.274016 16296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:12.274752 16294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:12.275030 16288 server_base.cc:1034] running on GCE node
I20240501 20:08:12.275735 16288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:12.276717 16288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:12.277946 16288 hybrid_clock.cc:648] HybridClock initialized: now 1714594092277910 us; error 50 us; skew 500 ppm
I20240501 20:08:12.278273 16288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:12.280824 16288 webserver.cc:460] Webserver started at http://127.13.251.129:41385/ using document root <none> and password file <none>
I20240501 20:08:12.281615 16288 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:12.281731 16288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:12.282058 16288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:12.284790 16288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b75e08cafe834ce580d6666f3a22bdfc"
format_stamp: "Formatted at 2024-05-01 20:08:12 on dist-test-slave-qswb"
I20240501 20:08:12.285518 16288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b75e08cafe834ce580d6666f3a22bdfc"
format_stamp: "Formatted at 2024-05-01 20:08:12 on dist-test-slave-qswb"
I20240501 20:08:12.291019 16288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 20:08:12.295029 16302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:12.296631 16288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 20:08:12.296787 16288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b75e08cafe834ce580d6666f3a22bdfc"
format_stamp: "Formatted at 2024-05-01 20:08:12 on dist-test-slave-qswb"
I20240501 20:08:12.297008 16288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:12.314615 16288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:12.315510 16288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:12.315846 16288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:12.316855 16288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:12.318414 16288 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:12.318499 16288 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:12.318588 16288 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:12.318660 16288 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:12.352277 16288 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.129:44281
I20240501 20:08:12.352330 16414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.129:44281 every 8 connection(s)
I20240501 20:08:12.353614 16288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:12.354014 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 16288
I20240501 20:08:12.354246 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:12.358089 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.130:0
--local_ip_for_outbound_sockets=127.13.251.130
--webserver_interface=127.13.251.130
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:12.369735 16415 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:12.370090 16415 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:12.370888 16415 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:12.372979 16229 ts_manager.cc:194] Registered new tserver with Master: b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:12.374533 16229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.129:59975
W20240501 20:08:12.691329 16419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:12.691679 16419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:12.691799 16419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:12.699440 16419 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:12.699623 16419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:12.699780 16419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.130
I20240501 20:08:12.709225 16419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.251.130
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.130
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:12.710609 16419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:12.712402 16419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:12.724503 16424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:12.725152 16427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:12.725731 16425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:12.725984 16419 server_base.cc:1034] running on GCE node
I20240501 20:08:12.726572 16419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:12.727595 16419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:12.728802 16419 hybrid_clock.cc:648] HybridClock initialized: now 1714594092728759 us; error 43 us; skew 500 ppm
I20240501 20:08:12.729144 16419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:12.731568 16419 webserver.cc:460] Webserver started at http://127.13.251.130:40149/ using document root <none> and password file <none>
I20240501 20:08:12.732338 16419 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:12.732452 16419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:12.732769 16419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:12.735455 16419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "63a2e807217a4182a7f7e5a1a105b376"
format_stamp: "Formatted at 2024-05-01 20:08:12 on dist-test-slave-qswb"
I20240501 20:08:12.736287 16419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "63a2e807217a4182a7f7e5a1a105b376"
format_stamp: "Formatted at 2024-05-01 20:08:12 on dist-test-slave-qswb"
I20240501 20:08:12.741591 16419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:12.745141 16433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:12.746654 16419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:12.746816 16419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "63a2e807217a4182a7f7e5a1a105b376"
format_stamp: "Formatted at 2024-05-01 20:08:12 on dist-test-slave-qswb"
I20240501 20:08:12.747032 16419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:12.762202 16419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:12.763106 16419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:12.763382 16419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:12.764410 16419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:12.765949 16419 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:12.766041 16419 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:12.766160 16419 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:12.766230 16419 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:12.799098 16419 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.130:41035
I20240501 20:08:12.799144 16545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.130:41035 every 8 connection(s)
I20240501 20:08:12.800591 16419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:12.809765 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 16419
I20240501 20:08:12.809934 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:12.813432 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.131:0
--local_ip_for_outbound_sockets=127.13.251.131
--webserver_interface=127.13.251.131
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:12.815665 16546 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:12.815994 16546 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:12.816776 16546 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:12.818308 16229 ts_manager.cc:194] Registered new tserver with Master: 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:12.819242 16229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.130:60639
W20240501 20:08:13.142282 16549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:13.142583 16549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:13.142673 16549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:13.150161 16549 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:13.150298 16549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:13.150460 16549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.131
I20240501 20:08:13.159649 16549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.251.131
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.131
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:13.160996 16549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:13.162765 16549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:13.174508 16554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:13.175143 16555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:13.175652 16557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:13.176307 16549 server_base.cc:1034] running on GCE node
I20240501 20:08:13.176867 16549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:13.177757 16549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:13.178953 16549 hybrid_clock.cc:648] HybridClock initialized: now 1714594093178909 us; error 41 us; skew 500 ppm
I20240501 20:08:13.179263 16549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:13.181517 16549 webserver.cc:460] Webserver started at http://127.13.251.131:46375/ using document root <none> and password file <none>
I20240501 20:08:13.182240 16549 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:13.182350 16549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:13.182653 16549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:13.185302 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6427cd62d4294811a5b4fd1cbd81d448"
format_stamp: "Formatted at 2024-05-01 20:08:13 on dist-test-slave-qswb"
I20240501 20:08:13.186053 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6427cd62d4294811a5b4fd1cbd81d448"
format_stamp: "Formatted at 2024-05-01 20:08:13 on dist-test-slave-qswb"
I20240501 20:08:13.191208 16549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:08:13.194530 16563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:13.195887 16549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:08:13.196033 16549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6427cd62d4294811a5b4fd1cbd81d448"
format_stamp: "Formatted at 2024-05-01 20:08:13 on dist-test-slave-qswb"
I20240501 20:08:13.196230 16549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:13.214900 16549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:13.215777 16549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:13.216053 16549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:13.217005 16549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:13.218501 16549 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:13.218593 16549 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:13.218711 16549 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:13.218780 16549 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:13.250577 16549 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.131:45387
I20240501 20:08:13.250633 16675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.131:45387 every 8 connection(s)
I20240501 20:08:13.251919 16549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:13.252671 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 16549
I20240501 20:08:13.252903 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:13.256485 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.132:0
--local_ip_for_outbound_sockets=127.13.251.132
--webserver_interface=127.13.251.132
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:13.265054 16676 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:13.265358 16676 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:13.266085 16676 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:13.267612 16229 ts_manager.cc:194] Registered new tserver with Master: 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131:45387)
I20240501 20:08:13.268863 16229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.131:47215
I20240501 20:08:13.377684 16415 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
W20240501 20:08:13.591789 16679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:13.592078 16679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:13.592171 16679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:13.599808 16679 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:13.599949 16679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:13.600069 16679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.132
I20240501 20:08:13.609599 16679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.13.251.132
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.132
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:13.610882 16679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:13.612643 16679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:13.624504 16685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:13.625051 16684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:13.625785 16687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:13.626075 16679 server_base.cc:1034] running on GCE node
I20240501 20:08:13.626675 16679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:13.627691 16679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:13.628897 16679 hybrid_clock.cc:648] HybridClock initialized: now 1714594093628838 us; error 51 us; skew 500 ppm
I20240501 20:08:13.629206 16679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:13.631501 16679 webserver.cc:460] Webserver started at http://127.13.251.132:35051/ using document root <none> and password file <none>
I20240501 20:08:13.632288 16679 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:13.632401 16679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:13.632712 16679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:13.635250 16679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "10d2f39dd2434b8e9b9d683defc79425"
format_stamp: "Formatted at 2024-05-01 20:08:13 on dist-test-slave-qswb"
I20240501 20:08:13.635993 16679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "10d2f39dd2434b8e9b9d683defc79425"
format_stamp: "Formatted at 2024-05-01 20:08:13 on dist-test-slave-qswb"
I20240501 20:08:13.640980 16679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240501 20:08:13.644275 16693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:13.645537 16679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:13.645676 16679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "10d2f39dd2434b8e9b9d683defc79425"
format_stamp: "Formatted at 2024-05-01 20:08:13 on dist-test-slave-qswb"
I20240501 20:08:13.645859 16679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:13.681564 16679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:13.682439 16679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:13.682736 16679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:13.683748 16679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:13.685220 16679 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:13.685315 16679 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:13.685431 16679 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:13.685503 16679 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:13.716415 16679 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.132:43563
I20240501 20:08:13.716487 16805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.132:43563 every 8 connection(s)
I20240501 20:08:13.717744 16679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:13.726580 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 16679
I20240501 20:08:13.726763 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:13.730185 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.133:0
--local_ip_for_outbound_sockets=127.13.251.133
--webserver_interface=127.13.251.133
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:13.730581 16806 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:13.730932 16806 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:13.731983 16806 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:13.733464 16229 ts_manager.cc:194] Registered new tserver with Master: 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132:43563)
I20240501 20:08:13.734244 16229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.132:36243
I20240501 20:08:13.821980 16546 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
W20240501 20:08:14.050612 16809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:14.050901 16809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:14.051045 16809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:14.058611 16809 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:14.058764 16809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:14.058933 16809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.133
I20240501 20:08:14.067744 16809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.13.251.133
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.133
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:14.069065 16809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:14.070749 16809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:14.081887 16814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:14.082468 16817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:14.083204 16815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:14.083945 16809 server_base.cc:1034] running on GCE node
I20240501 20:08:14.084496 16809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:14.085330 16809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:14.086517 16809 hybrid_clock.cc:648] HybridClock initialized: now 1714594094086485 us; error 23 us; skew 500 ppm
I20240501 20:08:14.086827 16809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:14.088891 16809 webserver.cc:460] Webserver started at http://127.13.251.133:33305/ using document root <none> and password file <none>
I20240501 20:08:14.089550 16809 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:14.089653 16809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:14.089958 16809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:14.092438 16809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "cc976bfc5fd643c8bb631f097245c65f"
format_stamp: "Formatted at 2024-05-01 20:08:14 on dist-test-slave-qswb"
I20240501 20:08:14.093138 16809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "cc976bfc5fd643c8bb631f097245c65f"
format_stamp: "Formatted at 2024-05-01 20:08:14 on dist-test-slave-qswb"
I20240501 20:08:14.097950 16809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:08:14.101154 16823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:14.102509 16809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:14.102645 16809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cc976bfc5fd643c8bb631f097245c65f"
format_stamp: "Formatted at 2024-05-01 20:08:14 on dist-test-slave-qswb"
I20240501 20:08:14.102833 16809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:14.123612 16809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:14.124464 16809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:14.124722 16809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:14.125726 16809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:14.127261 16809 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:14.127349 16809 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:14.127461 16809 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:14.127524 16809 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:14.158993 16809 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.133:41735
I20240501 20:08:14.159041 16935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.133:41735 every 8 connection(s)
I20240501 20:08:14.160323 16809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:14.168556 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 16809
I20240501 20:08:14.168757 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:14.171818 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.134:0
--local_ip_for_outbound_sockets=127.13.251.134
--webserver_interface=127.13.251.134
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:14.173301 16936 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:14.173655 16936 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:14.174623 16936 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:14.176100 16229 ts_manager.cc:194] Registered new tserver with Master: cc976bfc5fd643c8bb631f097245c65f (127.13.251.133:41735)
I20240501 20:08:14.176813 16229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.133:49469
I20240501 20:08:14.271582 16676 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
W20240501 20:08:14.510818 16939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:14.511113 16939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:14.511207 16939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:14.518755 16939 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:14.518910 16939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:14.519086 16939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.134
I20240501 20:08:14.528929 16939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.13.251.134
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.134
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:14.530387 16939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:14.532347 16939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:14.544617 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:14.545312 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:14.545915 16939 server_base.cc:1034] running on GCE node
W20240501 20:08:14.546480 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:14.547118 16939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:14.548081 16939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:14.549301 16939 hybrid_clock.cc:648] HybridClock initialized: now 1714594094549237 us; error 37 us; skew 500 ppm
I20240501 20:08:14.549643 16939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:14.552022 16939 webserver.cc:460] Webserver started at http://127.13.251.134:43885/ using document root <none> and password file <none>
I20240501 20:08:14.553028 16939 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:14.553133 16939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:14.553413 16939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:14.556118 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "474b2d91e93f46be855e2bf1243bf0c8"
format_stamp: "Formatted at 2024-05-01 20:08:14 on dist-test-slave-qswb"
I20240501 20:08:14.556890 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "474b2d91e93f46be855e2bf1243bf0c8"
format_stamp: "Formatted at 2024-05-01 20:08:14 on dist-test-slave-qswb"
I20240501 20:08:14.562239 16939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:14.565752 16953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:14.567062 16939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:14.567238 16939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "474b2d91e93f46be855e2bf1243bf0c8"
format_stamp: "Formatted at 2024-05-01 20:08:14 on dist-test-slave-qswb"
I20240501 20:08:14.567433 16939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:14.581388 16939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:14.582222 16939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:14.582494 16939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:14.583424 16939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:14.584864 16939 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:14.584962 16939 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:14.585076 16939 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:14.585155 16939 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:14.617053 16939 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.134:40765
I20240501 20:08:14.617096 17065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.134:40765 every 8 connection(s)
I20240501 20:08:14.618347 16939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:08:14.620777 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 16939
I20240501 20:08:14.620946 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:08:14.624310 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.135:0
--local_ip_for_outbound_sockets=127.13.251.135
--webserver_interface=127.13.251.135
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:14.632856 17066 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:14.633174 17066 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:14.633901 17066 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:14.635279 16229 ts_manager.cc:194] Registered new tserver with Master: 474b2d91e93f46be855e2bf1243bf0c8 (127.13.251.134:40765)
I20240501 20:08:14.636137 16229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.134:56039
I20240501 20:08:14.736821 16806 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
W20240501 20:08:14.964428 17069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:14.964759 17069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:14.964908 17069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:14.972966 17069 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:14.973119 17069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:14.973281 17069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.135
I20240501 20:08:14.982559 17069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.13.251.135
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.135
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:14.984045 17069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:14.985749 17069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:14.998806 17077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:14.998975 17074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:14.999452 17075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:15.000329 17069 server_base.cc:1034] running on GCE node
I20240501 20:08:15.001003 17069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:15.002151 17069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:15.003839 17069 hybrid_clock.cc:648] HybridClock initialized: now 1714594095003315 us; error 50 us; skew 500 ppm
I20240501 20:08:15.004173 17069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:15.006520 17069 webserver.cc:460] Webserver started at http://127.13.251.135:38659/ using document root <none> and password file <none>
I20240501 20:08:15.007311 17069 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:15.007436 17069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:15.007786 17069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:15.010488 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "3c7f7a1988b749d6a2bfa4d374e87e69"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-qswb"
I20240501 20:08:15.011333 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "3c7f7a1988b749d6a2bfa4d374e87e69"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-qswb"
I20240501 20:08:15.016842 17069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 20:08:15.020309 17083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:15.021672 17069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:15.021828 17069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "3c7f7a1988b749d6a2bfa4d374e87e69"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-qswb"
I20240501 20:08:15.022028 17069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:15.052022 17069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:15.052917 17069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:15.053184 17069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:15.054177 17069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:15.055891 17069 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:15.055986 17069 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:15.056102 17069 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:15.056172 17069 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:15.094841 17069 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.135:34297
I20240501 20:08:15.094902 17195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.135:34297 every 8 connection(s)
I20240501 20:08:15.096232 17069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:08:15.104967 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 17069
I20240501 20:08:15.105168 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:08:15.108996 17196 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:15.109297 17196 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:15.110493 17196 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:15.111820 16228 ts_manager.cc:194] Registered new tserver with Master: 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135:34297)
I20240501 20:08:15.112725 16228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.135:37859
I20240501 20:08:15.121287 14318 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:08:15.139490 16228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51550:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:08:15.173385 17001 tablet_service.cc:1467] Processing CreateTablet for tablet 09d70c138c8240b394fa6bdb0d3018d9 (DEFAULT_TABLE table=TestTable [id=e86de4c1a9a64cddb6a69deb6b0d4995]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:15.174796 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d70c138c8240b394fa6bdb0d3018d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:15.179116 16936 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
I20240501 20:08:15.186376 17210 tablet_bootstrap.cc:492] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Bootstrap starting.
I20240501 20:08:15.189594 17210 tablet_bootstrap.cc:654] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:15.190951 17210 log.cc:824] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:15.193545 17210 tablet_bootstrap.cc:492] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: No bootstrap required, opened a new log
I20240501 20:08:15.193817 17210 ts_tablet_manager.cc:1367] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 20:08:15.199965 17210 raft_consensus.cc:348] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:15.200316 17210 raft_consensus.cc:374] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:15.200421 17210 raft_consensus.cc:729] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474b2d91e93f46be855e2bf1243bf0c8, State: Initialized, Role: FOLLOWER
I20240501 20:08:15.201545 17210 consensus_queue.cc:260] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:15.201741 17210 raft_consensus.cc:388] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:15.201870 17210 raft_consensus.cc:482] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:15.202024 17210 raft_consensus.cc:3037] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:15.204109 17210 raft_consensus.cc:504] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:15.204604 17210 leader_election.cc:302] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 474b2d91e93f46be855e2bf1243bf0c8; no voters:
I20240501 20:08:15.205183 17210 leader_election.cc:288] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:15.205338 17212 raft_consensus.cc:2781] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:15.206032 17212 raft_consensus.cc:686] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 LEADER]: Becoming Leader. State: Replica: 474b2d91e93f46be855e2bf1243bf0c8, State: Running, Role: LEADER
I20240501 20:08:15.206408 17066 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
I20240501 20:08:15.206655 17212 consensus_queue.cc:237] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:15.207165 17210 ts_tablet_manager.cc:1398] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20240501 20:08:15.212144 16229 catalog_manager.cc:5484] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 474b2d91e93f46be855e2bf1243bf0c8 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "474b2d91e93f46be855e2bf1243bf0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } } }
I20240501 20:08:15.230010 14318 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:08:15.232065 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b75e08cafe834ce580d6666f3a22bdfc to finish bootstrapping
I20240501 20:08:15.241988 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 63a2e807217a4182a7f7e5a1a105b376 to finish bootstrapping
I20240501 20:08:15.251369 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6427cd62d4294811a5b4fd1cbd81d448 to finish bootstrapping
I20240501 20:08:15.260758 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 10d2f39dd2434b8e9b9d683defc79425 to finish bootstrapping
I20240501 20:08:15.269881 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cc976bfc5fd643c8bb631f097245c65f to finish bootstrapping
I20240501 20:08:15.278874 14318 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 474b2d91e93f46be855e2bf1243bf0c8 to finish bootstrapping
I20240501 20:08:15.286099 14318 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3c7f7a1988b749d6a2bfa4d374e87e69 to finish bootstrapping
I20240501 20:08:15.295398 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 16549
I20240501 20:08:15.296460 16671 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:15.378769 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 16549
I20240501 20:08:15.395457 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 16679
I20240501 20:08:15.396574 16801 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:15.476601 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 16679
I20240501 20:08:15.492605 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 16809
I20240501 20:08:15.493778 16931 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:15.575109 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 16809
I20240501 20:08:15.590972 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 16939
I20240501 20:08:15.592057 17061 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:15.677927 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 16939
I20240501 20:08:15.694581 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 17069
I20240501 20:08:15.695633 17191 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:15.774410 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 17069
I20240501 20:08:19.543431 16229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51550:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:19.563490 16481 tablet_service.cc:1467] Processing CreateTablet for tablet 44f3b394dd644f1da3821d53dd7c735b (DEFAULT_TABLE table=rebalance_test_table_0 [id=850cf659e4ff4f46aa0f6b6664d00a27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:19.565028 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f3b394dd644f1da3821d53dd7c735b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:19.564740 16349 tablet_service.cc:1467] Processing CreateTablet for tablet 699e4265f7d14e47b55ab17594554f77 (DEFAULT_TABLE table=rebalance_test_table_0 [id=850cf659e4ff4f46aa0f6b6664d00a27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:19.564476 16350 tablet_service.cc:1467] Processing CreateTablet for tablet f522ede49dac4f7193158c6bca98ed26 (DEFAULT_TABLE table=rebalance_test_table_0 [id=850cf659e4ff4f46aa0f6b6664d00a27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:19.566045 16349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699e4265f7d14e47b55ab17594554f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:19.566511 16350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f522ede49dac4f7193158c6bca98ed26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:19.576532 17237 tablet_bootstrap.cc:492] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376: Bootstrap starting.
I20240501 20:08:19.579466 17238 tablet_bootstrap.cc:492] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: Bootstrap starting.
I20240501 20:08:19.580312 17237 tablet_bootstrap.cc:654] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:19.581669 17237 log.cc:824] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:19.582470 17238 tablet_bootstrap.cc:654] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:19.583496 17238 log.cc:824] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:19.584540 17237 tablet_bootstrap.cc:492] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376: No bootstrap required, opened a new log
I20240501 20:08:19.584810 17237 ts_tablet_manager.cc:1367] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20240501 20:08:19.586033 17238 tablet_bootstrap.cc:492] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: No bootstrap required, opened a new log
I20240501 20:08:19.586278 17238 ts_tablet_manager.cc:1367] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:08:19.591284 17237 raft_consensus.cc:348] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:19.591703 17237 raft_consensus.cc:374] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:19.591820 17237 raft_consensus.cc:729] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Initialized, Role: FOLLOWER
I20240501 20:08:19.592516 17237 consensus_queue.cc:260] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:19.592720 17237 raft_consensus.cc:388] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:19.592870 17237 raft_consensus.cc:482] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:19.593020 17237 raft_consensus.cc:3037] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:19.592953 17238 raft_consensus.cc:348] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:19.593324 17238 raft_consensus.cc:374] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:19.593422 17238 raft_consensus.cc:729] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Initialized, Role: FOLLOWER
I20240501 20:08:19.594022 17238 consensus_queue.cc:260] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:19.594228 17238 raft_consensus.cc:388] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:19.594352 17238 raft_consensus.cc:482] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:19.594508 17238 raft_consensus.cc:3037] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:19.595345 17237 raft_consensus.cc:504] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:19.595849 17237 leader_election.cc:302] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376; no voters:
I20240501 20:08:19.596527 17237 leader_election.cc:288] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:19.596661 17241 raft_consensus.cc:2781] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:19.596602 17238 raft_consensus.cc:504] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:19.597184 17238 leader_election.cc:302] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:19.598505 17237 ts_tablet_manager.cc:1398] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20240501 20:08:19.599170 17241 raft_consensus.cc:686] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Leader. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:19.599716 17241 consensus_queue.cc:237] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:19.599953 17238 leader_election.cc:288] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:19.600047 17244 raft_consensus.cc:2781] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:19.602419 17238 ts_tablet_manager.cc:1398] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: Time spent starting tablet: real 0.016s user 0.005s sys 0.010s
I20240501 20:08:19.602977 17238 tablet_bootstrap.cc:492] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: Bootstrap starting.
I20240501 20:08:19.605443 17238 tablet_bootstrap.cc:654] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:19.606712 16229 catalog_manager.cc:5484] T 44f3b394dd644f1da3821d53dd7c735b P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } }
I20240501 20:08:19.606943 17244 raft_consensus.cc:686] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Leader. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:19.607674 17244 consensus_queue.cc:237] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
W20240501 20:08:19.612633 16416 tablet.cc:2369] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:19.613080 17238 tablet_bootstrap.cc:492] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: No bootstrap required, opened a new log
I20240501 20:08:19.613241 17238 ts_tablet_manager.cc:1367] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240501 20:08:19.614426 17238 raft_consensus.cc:348] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:19.614642 17238 raft_consensus.cc:374] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:19.614785 17238 raft_consensus.cc:729] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Initialized, Role: FOLLOWER
I20240501 20:08:19.615118 17238 consensus_queue.cc:260] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:19.615295 17238 raft_consensus.cc:388] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:19.615387 17238 raft_consensus.cc:482] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:19.615514 17238 raft_consensus.cc:3037] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:19.618263 16229 catalog_manager.cc:5484] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129). New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } }
I20240501 20:08:19.618896 17238 raft_consensus.cc:504] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:19.619345 17238 leader_election.cc:302] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:19.619827 17246 raft_consensus.cc:2781] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:19.619980 17246 raft_consensus.cc:686] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Leader. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:19.620301 17246 consensus_queue.cc:237] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:19.621330 17238 leader_election.cc:288] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:19.622026 17238 ts_tablet_manager.cc:1398] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:08:19.625768 16228 catalog_manager.cc:5484] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129). New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } }
W20240501 20:08:19.990480 17254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:19.990855 17254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:19.998407 17254 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 130.836 ms
time per row: 0.0654181 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:08:21.206218 16228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51550:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:21.215027 16481 tablet_service.cc:1467] Processing CreateTablet for tablet fb81d991e8a3455db9f94b6ef07b86ee (DEFAULT_TABLE table=rebalance_test_table_1 [id=373549282c7d4d6c98c887b22195d023]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:21.215135 16480 tablet_service.cc:1467] Processing CreateTablet for tablet 317b376708eb4b268b93fb8d1c15258a (DEFAULT_TABLE table=rebalance_test_table_1 [id=373549282c7d4d6c98c887b22195d023]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:21.215366 16349 tablet_service.cc:1467] Processing CreateTablet for tablet 70db4cb78b6f4256b4752601bc730808 (DEFAULT_TABLE table=rebalance_test_table_1 [id=373549282c7d4d6c98c887b22195d023]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:21.215509 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb81d991e8a3455db9f94b6ef07b86ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:21.215783 16480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 317b376708eb4b268b93fb8d1c15258a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:21.215785 16349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70db4cb78b6f4256b4752601bc730808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:21.221019 17303 tablet_bootstrap.cc:492] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc: Bootstrap starting.
I20240501 20:08:21.222472 17304 tablet_bootstrap.cc:492] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: Bootstrap starting.
I20240501 20:08:21.222898 17303 tablet_bootstrap.cc:654] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:21.224247 17304 tablet_bootstrap.cc:654] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:21.224860 17303 tablet_bootstrap.cc:492] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc: No bootstrap required, opened a new log
I20240501 20:08:21.225006 17303 ts_tablet_manager.cc:1367] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:08:21.225865 17303 raft_consensus.cc:348] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:21.226056 17303 raft_consensus.cc:374] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:21.226161 17303 raft_consensus.cc:729] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Initialized, Role: FOLLOWER
I20240501 20:08:21.226449 17304 tablet_bootstrap.cc:492] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: No bootstrap required, opened a new log
I20240501 20:08:21.226399 17303 consensus_queue.cc:260] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:21.226605 17304 ts_tablet_manager.cc:1367] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 20:08:21.226601 17303 raft_consensus.cc:388] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:21.226681 17303 raft_consensus.cc:482] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:21.226800 17303 raft_consensus.cc:3037] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:21.227576 17304 raft_consensus.cc:348] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:21.227748 17304 raft_consensus.cc:374] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:21.227838 17304 raft_consensus.cc:729] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Initialized, Role: FOLLOWER
I20240501 20:08:21.228092 17304 consensus_queue.cc:260] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:21.228272 17304 raft_consensus.cc:388] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:21.228348 17304 raft_consensus.cc:482] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:21.228454 17304 raft_consensus.cc:3037] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:21.229552 17303 raft_consensus.cc:504] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:21.229820 17303 leader_election.cc:302] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:21.230273 17303 leader_election.cc:288] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:21.230435 17307 raft_consensus.cc:2781] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:21.231130 17303 ts_tablet_manager.cc:1398] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:08:21.231215 17307 raft_consensus.cc:686] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Leader. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:21.231366 17304 raft_consensus.cc:504] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:21.231484 17307 consensus_queue.cc:237] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:21.231623 17304 leader_election.cc:302] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376; no voters:
I20240501 20:08:21.232054 17304 leader_election.cc:288] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:21.232183 17310 raft_consensus.cc:2781] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:21.233541 17310 raft_consensus.cc:686] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Leader. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:21.233842 17310 consensus_queue.cc:237] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:21.234102 16228 catalog_manager.cc:5484] T 70db4cb78b6f4256b4752601bc730808 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129). New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } }
I20240501 20:08:21.237582 17304 ts_tablet_manager.cc:1398] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20240501 20:08:21.237941 17304 tablet_bootstrap.cc:492] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: Bootstrap starting.
I20240501 20:08:21.239570 16225 catalog_manager.cc:5484] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } }
I20240501 20:08:21.240475 17304 tablet_bootstrap.cc:654] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:21.245198 17304 tablet_bootstrap.cc:492] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: No bootstrap required, opened a new log
I20240501 20:08:21.245471 17304 ts_tablet_manager.cc:1367] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:08:21.246585 17304 raft_consensus.cc:348] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:21.246855 17304 raft_consensus.cc:374] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:21.246959 17304 raft_consensus.cc:729] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Initialized, Role: FOLLOWER
I20240501 20:08:21.247282 17304 consensus_queue.cc:260] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:21.247509 17304 raft_consensus.cc:388] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:21.247685 17304 raft_consensus.cc:482] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:21.247812 17304 raft_consensus.cc:3037] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:21.252028 17304 raft_consensus.cc:504] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:21.252341 17304 leader_election.cc:302] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376; no voters:
I20240501 20:08:21.252578 17304 leader_election.cc:288] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:21.252707 17312 raft_consensus.cc:2781] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:21.252915 17312 raft_consensus.cc:686] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Leader. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:21.253090 17304 ts_tablet_manager.cc:1398] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:08:21.253221 17312 consensus_queue.cc:237] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:21.257047 16225 catalog_manager.cc:5484] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } }
W20240501 20:08:21.613710 17319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:21.614071 17319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:21.622054 17319 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 132.557 ms
time per row: 0.0662786 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:08:22.853554 16225 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51550:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:22.862821 16349 tablet_service.cc:1467] Processing CreateTablet for tablet 609385e741e04820bb903f2664108ff2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=357f26ae29a643e3b064aeb9bcfc24ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:22.863164 16480 tablet_service.cc:1467] Processing CreateTablet for tablet 33eb5bad3a7e48559804a50b09085802 (DEFAULT_TABLE table=rebalance_test_table_2 [id=357f26ae29a643e3b064aeb9bcfc24ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:22.863332 16349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609385e741e04820bb903f2664108ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:22.863606 16480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33eb5bad3a7e48559804a50b09085802. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:22.863643 16350 tablet_service.cc:1467] Processing CreateTablet for tablet 3acfb78e3127445db565fc5ceaca8b9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=357f26ae29a643e3b064aeb9bcfc24ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:22.864034 16350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acfb78e3127445db565fc5ceaca8b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:22.869374 17364 tablet_bootstrap.cc:492] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: Bootstrap starting.
I20240501 20:08:22.870661 17365 tablet_bootstrap.cc:492] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376: Bootstrap starting.
I20240501 20:08:22.871282 17364 tablet_bootstrap.cc:654] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:22.872056 17365 tablet_bootstrap.cc:654] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:22.874884 17365 tablet_bootstrap.cc:492] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376: No bootstrap required, opened a new log
I20240501 20:08:22.875051 17365 ts_tablet_manager.cc:1367] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:08:22.875144 17364 tablet_bootstrap.cc:492] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: No bootstrap required, opened a new log
I20240501 20:08:22.875298 17364 ts_tablet_manager.cc:1367] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:08:22.875968 17364 raft_consensus.cc:348] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:22.875957 17365 raft_consensus.cc:348] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:22.876139 17364 raft_consensus.cc:374] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:22.876142 17365 raft_consensus.cc:374] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:22.876230 17364 raft_consensus.cc:729] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Initialized, Role: FOLLOWER
I20240501 20:08:22.876230 17365 raft_consensus.cc:729] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Initialized, Role: FOLLOWER
I20240501 20:08:22.876492 17364 consensus_queue.cc:260] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:22.876467 17365 consensus_queue.cc:260] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:22.876652 17364 raft_consensus.cc:388] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:22.876662 17365 raft_consensus.cc:388] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:22.876731 17364 raft_consensus.cc:482] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:22.876735 17365 raft_consensus.cc:482] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:22.876838 17365 raft_consensus.cc:3037] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:22.876847 17364 raft_consensus.cc:3037] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:22.977715 17364 raft_consensus.cc:504] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:22.977712 17365 raft_consensus.cc:504] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:22.978096 17364 leader_election.cc:302] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:22.978099 17365 leader_election.cc:302] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376; no voters:
I20240501 20:08:22.978665 17364 leader_election.cc:288] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:22.978958 17365 leader_election.cc:288] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:22.979140 17368 raft_consensus.cc:2781] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:22.979429 17369 raft_consensus.cc:2781] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:22.983268 17364 ts_tablet_manager.cc:1398] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: Time spent starting tablet: real 0.108s user 0.008s sys 0.000s
I20240501 20:08:22.983469 17369 raft_consensus.cc:686] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Leader. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:22.983633 17364 tablet_bootstrap.cc:492] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: Bootstrap starting.
I20240501 20:08:22.983894 17369 consensus_queue.cc:237] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:22.985714 17364 tablet_bootstrap.cc:654] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:22.986233 17365 ts_tablet_manager.cc:1398] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376: Time spent starting tablet: real 0.111s user 0.006s sys 0.007s
I20240501 20:08:22.986248 17368 raft_consensus.cc:686] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Leader. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:22.986685 17368 consensus_queue.cc:237] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:22.988014 17364 tablet_bootstrap.cc:492] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: No bootstrap required, opened a new log
I20240501 20:08:22.988996 17364 ts_tablet_manager.cc:1367] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 20:08:22.989114 16225 catalog_manager.cc:5484] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129). New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } }
I20240501 20:08:22.990075 17364 raft_consensus.cc:348] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:22.990278 17364 raft_consensus.cc:374] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:22.990366 17364 raft_consensus.cc:729] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Initialized, Role: FOLLOWER
I20240501 20:08:22.990700 17364 consensus_queue.cc:260] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:22.990904 17364 raft_consensus.cc:388] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:22.991007 17364 raft_consensus.cc:482] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:22.991112 17364 raft_consensus.cc:3037] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:22.991021 16226 catalog_manager.cc:5484] T 33eb5bad3a7e48559804a50b09085802 P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } }
I20240501 20:08:23.081140 17364 raft_consensus.cc:504] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:23.081499 17364 leader_election.cc:302] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:23.081743 17364 leader_election.cc:288] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:23.081827 17369 raft_consensus.cc:2781] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:23.081977 17369 raft_consensus.cc:686] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Leader. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:23.082295 17364 ts_tablet_manager.cc:1398] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: Time spent starting tablet: real 0.093s user 0.001s sys 0.004s
I20240501 20:08:23.082270 17369 consensus_queue.cc:237] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:23.084648 16226 catalog_manager.cc:5484] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129). New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } }
W20240501 20:08:23.460803 17380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:23.461189 17380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:23.469329 17380 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 134.372 ms
time per row: 0.0671861 ms
I20240501 20:08:23.753577 16225 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51550:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:23.762545 16480 tablet_service.cc:1467] Processing CreateTablet for tablet 608ac70c00014df693e8bd48925af0c3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4e128faf8de54db1923732a8c5a70f5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:23.763046 16480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608ac70c00014df693e8bd48925af0c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:23.763105 16349 tablet_service.cc:1467] Processing CreateTablet for tablet 99cfaa6da58247f0b477bf2cbe1aea51 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4e128faf8de54db1923732a8c5a70f5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:23.763463 16349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99cfaa6da58247f0b477bf2cbe1aea51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:23.762603 16481 tablet_service.cc:1467] Processing CreateTablet for tablet 0b380bf3c85748cfb4843ae21eadc74b (DEFAULT_TABLE table=rebalance_test_table_3 [id=4e128faf8de54db1923732a8c5a70f5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:23.768054 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b380bf3c85748cfb4843ae21eadc74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:23.768121 17427 tablet_bootstrap.cc:492] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc: Bootstrap starting.
I20240501 20:08:23.769469 17428 tablet_bootstrap.cc:492] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: Bootstrap starting.
I20240501 20:08:23.770001 17427 tablet_bootstrap.cc:654] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:23.771998 17428 tablet_bootstrap.cc:654] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:23.772487 17427 tablet_bootstrap.cc:492] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc: No bootstrap required, opened a new log
I20240501 20:08:23.772632 17427 ts_tablet_manager.cc:1367] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:08:23.773420 17427 raft_consensus.cc:348] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:23.773608 17427 raft_consensus.cc:374] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:23.773691 17427 raft_consensus.cc:729] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Initialized, Role: FOLLOWER
I20240501 20:08:23.773936 17427 consensus_queue.cc:260] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:23.774118 17427 raft_consensus.cc:388] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:23.774181 17427 raft_consensus.cc:482] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:23.774276 17427 raft_consensus.cc:3037] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:23.775856 17428 tablet_bootstrap.cc:492] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: No bootstrap required, opened a new log
I20240501 20:08:23.775998 17428 ts_tablet_manager.cc:1367] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:08:23.776605 17428 raft_consensus.cc:348] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:23.776741 17428 raft_consensus.cc:374] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:23.776823 17428 raft_consensus.cc:729] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Initialized, Role: FOLLOWER
I20240501 20:08:23.777048 17428 consensus_queue.cc:260] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:23.777189 17428 raft_consensus.cc:388] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:23.777261 17428 raft_consensus.cc:482] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:23.777369 17428 raft_consensus.cc:3037] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:23.793772 17427 raft_consensus.cc:504] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:23.793772 17428 raft_consensus.cc:504] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:23.794160 17428 leader_election.cc:302] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376; no voters:
I20240501 20:08:23.794430 17427 leader_election.cc:302] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:23.794504 17412 raft_consensus.cc:2781] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:23.794757 17417 raft_consensus.cc:2781] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:23.794912 17417 raft_consensus.cc:686] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Leader. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:23.795339 17417 consensus_queue.cc:237] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:23.801072 17427 leader_election.cc:288] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:23.802194 17427 ts_tablet_manager.cc:1398] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20240501 20:08:23.803452 16225 catalog_manager.cc:5484] T 99cfaa6da58247f0b477bf2cbe1aea51 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129). New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } }
I20240501 20:08:23.804176 17412 raft_consensus.cc:686] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Leader. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:23.804540 17412 consensus_queue.cc:237] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:23.805186 17428 leader_election.cc:288] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:23.805698 17428 ts_tablet_manager.cc:1398] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20240501 20:08:23.806175 17428 tablet_bootstrap.cc:492] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: Bootstrap starting.
I20240501 20:08:23.808318 16225 catalog_manager.cc:5484] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } }
I20240501 20:08:23.808976 17428 tablet_bootstrap.cc:654] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:23.817989 17428 tablet_bootstrap.cc:492] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: No bootstrap required, opened a new log
I20240501 20:08:23.818264 17428 ts_tablet_manager.cc:1367] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240501 20:08:23.819227 17428 raft_consensus.cc:348] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:23.819463 17428 raft_consensus.cc:374] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:23.819577 17428 raft_consensus.cc:729] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Initialized, Role: FOLLOWER
I20240501 20:08:23.819873 17428 consensus_queue.cc:260] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:23.820063 17428 raft_consensus.cc:388] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:23.820179 17428 raft_consensus.cc:482] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:23.820281 17428 raft_consensus.cc:3037] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:23.894244 17428 raft_consensus.cc:504] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:23.894596 17428 leader_election.cc:302] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376; no voters:
I20240501 20:08:23.894845 17428 leader_election.cc:288] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:23.894934 17412 raft_consensus.cc:2781] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:23.895449 17428 ts_tablet_manager.cc:1398] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: Time spent starting tablet: real 0.077s user 0.002s sys 0.002s
I20240501 20:08:23.895849 17412 raft_consensus.cc:686] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Leader. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:23.896138 17412 consensus_queue.cc:237] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:23.899413 16225 catalog_manager.cc:5484] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } }
W20240501 20:08:24.264300 17437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:24.264653 17437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:24.273145 17437 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 149.856 ms
time per row: 0.0749278 ms
I20240501 20:08:24.578387 16225 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51550:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:24.587610 16349 tablet_service.cc:1467] Processing CreateTablet for tablet 08f747bc44c94917920cd5b84a4ac356 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4083fed5a42341cb8f80a011df1f5d16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:24.587786 16481 tablet_service.cc:1467] Processing CreateTablet for tablet eda61e86d34d4ee687006c3cae7a51bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=4083fed5a42341cb8f80a011df1f5d16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:24.588138 16349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f747bc44c94917920cd5b84a4ac356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:24.588253 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda61e86d34d4ee687006c3cae7a51bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:24.592274 16350 tablet_service.cc:1467] Processing CreateTablet for tablet c9e2c42ca67f4fea9e8dc724658c2090 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4083fed5a42341cb8f80a011df1f5d16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:24.592701 16350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e2c42ca67f4fea9e8dc724658c2090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:24.596351 17483 tablet_bootstrap.cc:492] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: Bootstrap starting.
I20240501 20:08:24.597458 17484 tablet_bootstrap.cc:492] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: Bootstrap starting.
I20240501 20:08:24.598649 17483 tablet_bootstrap.cc:654] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:24.599464 17484 tablet_bootstrap.cc:654] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:24.601084 17483 tablet_bootstrap.cc:492] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: No bootstrap required, opened a new log
I20240501 20:08:24.601258 17483 ts_tablet_manager.cc:1367] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:08:24.601660 17484 tablet_bootstrap.cc:492] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: No bootstrap required, opened a new log
I20240501 20:08:24.601835 17484 ts_tablet_manager.cc:1367] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:08:24.602077 17483 raft_consensus.cc:348] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:24.602224 17483 raft_consensus.cc:374] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:24.602305 17483 raft_consensus.cc:729] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Initialized, Role: FOLLOWER
I20240501 20:08:24.602557 17483 consensus_queue.cc:260] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:24.602614 17484 raft_consensus.cc:348] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:24.602711 17483 raft_consensus.cc:388] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:24.602788 17484 raft_consensus.cc:374] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:24.602787 17483 raft_consensus.cc:482] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:24.602877 17484 raft_consensus.cc:729] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Initialized, Role: FOLLOWER
I20240501 20:08:24.602896 17483 raft_consensus.cc:3037] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:24.603103 17484 consensus_queue.cc:260] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:24.603271 17484 raft_consensus.cc:388] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:24.603348 17484 raft_consensus.cc:482] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:24.603453 17484 raft_consensus.cc:3037] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:24.615733 17484 raft_consensus.cc:504] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:24.615780 17483 raft_consensus.cc:504] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:24.616068 17484 leader_election.cc:302] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:24.616120 17483 leader_election.cc:302] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376; no voters:
I20240501 20:08:24.616283 17484 leader_election.cc:288] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:24.616348 17483 leader_election.cc:288] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:24.616415 17469 raft_consensus.cc:2781] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:24.616650 17473 raft_consensus.cc:2781] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:24.616803 17473 raft_consensus.cc:686] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Leader. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:24.616948 17483 ts_tablet_manager.cc:1398] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: Time spent starting tablet: real 0.016s user 0.002s sys 0.003s
I20240501 20:08:24.617173 17473 consensus_queue.cc:237] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:24.622453 17484 ts_tablet_manager.cc:1398] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: Time spent starting tablet: real 0.020s user 0.009s sys 0.000s
I20240501 20:08:24.622521 17469 raft_consensus.cc:686] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Leader. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:24.622912 17484 tablet_bootstrap.cc:492] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: Bootstrap starting.
I20240501 20:08:24.623724 17469 consensus_queue.cc:237] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:24.624953 17484 tablet_bootstrap.cc:654] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:24.624928 16225 catalog_manager.cc:5484] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } }
I20240501 20:08:24.626685 16221 catalog_manager.cc:5484] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129). New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } }
I20240501 20:08:24.627660 17484 tablet_bootstrap.cc:492] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: No bootstrap required, opened a new log
I20240501 20:08:24.627830 17484 ts_tablet_manager.cc:1367] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:24.628612 17484 raft_consensus.cc:348] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:24.628819 17484 raft_consensus.cc:374] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:24.628912 17484 raft_consensus.cc:729] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Initialized, Role: FOLLOWER
I20240501 20:08:24.629593 17484 consensus_queue.cc:260] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:24.629747 17484 raft_consensus.cc:388] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:24.629859 17484 raft_consensus.cc:482] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:24.630016 17484 raft_consensus.cc:3037] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:24.719449 17484 raft_consensus.cc:504] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:24.719841 17484 leader_election.cc:302] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:24.720108 17484 leader_election.cc:288] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:24.720263 17469 raft_consensus.cc:2781] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:24.720515 17469 raft_consensus.cc:686] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Leader. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:24.720599 17484 ts_tablet_manager.cc:1398] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: Time spent starting tablet: real 0.093s user 0.005s sys 0.000s
I20240501 20:08:24.721184 17469 consensus_queue.cc:237] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:24.724006 16225 catalog_manager.cc:5484] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129). New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } }
W20240501 20:08:25.118120 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:25.118464 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:25.126391 17494 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 137.799 ms
time per row: 0.0688996 ms
I20240501 20:08:25.420447 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.131:45387
--local_ip_for_outbound_sockets=127.13.251.131
--tserver_master_addrs=127.13.251.190:43847
--webserver_port=46375
--webserver_interface=127.13.251.131
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:08:25.770661 17538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:25.770980 17538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:25.771071 17538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:25.778857 17538 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:25.778995 17538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:25.779119 17538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.131
I20240501 20:08:25.794804 17538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.131:45387
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.251.131
--webserver_port=46375
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.131
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:25.796767 17538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:25.799407 17538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:25.826440 17543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:25.827241 17544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:25.828331 17546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:25.829011 17538 server_base.cc:1034] running on GCE node
I20240501 20:08:25.829942 17538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:25.831514 17538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:25.832914 17538 hybrid_clock.cc:648] HybridClock initialized: now 1714594105832847 us; error 79 us; skew 500 ppm
I20240501 20:08:25.833405 17538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:25.836992 17538 webserver.cc:460] Webserver started at http://127.13.251.131:46375/ using document root <none> and password file <none>
I20240501 20:08:25.838368 17538 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:25.838549 17538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:25.847334 17538 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:08:25.851677 17552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.853816 17538 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 20:08:25.854027 17538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6427cd62d4294811a5b4fd1cbd81d448"
format_stamp: "Formatted at 2024-05-01 20:08:13 on dist-test-slave-qswb"
I20240501 20:08:25.855052 17538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:25.885841 17538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:25.886772 17538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:25.887089 17538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:25.888217 17538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:25.889880 17538 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:25.889966 17538 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.890117 17538 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:25.890180 17538 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:25.926600 17538 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.131:45387
I20240501 20:08:25.926630 17664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.131:45387 every 8 connection(s)
I20240501 20:08:25.928315 17538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:25.937183 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 17538
I20240501 20:08:25.938031 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.132:43563
--local_ip_for_outbound_sockets=127.13.251.132
--tserver_master_addrs=127.13.251.190:43847
--webserver_port=35051
--webserver_interface=127.13.251.132
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:25.943353 17665 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:25.943773 17665 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:25.944676 17665 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:25.946457 16221 ts_manager.cc:194] Re-registered known tserver with Master: 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131:45387)
I20240501 20:08:25.947767 16221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.131:58943
W20240501 20:08:26.292781 17669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:26.293092 17669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:26.293188 17669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:26.300849 17669 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:26.300989 17669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:26.301162 17669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.132
I20240501 20:08:26.310531 17669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.132:43563
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.13.251.132
--webserver_port=35051
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.132
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:26.311985 17669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:26.313867 17669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:26.326843 17674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:26.327919 17675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:26.328866 17677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:26.329721 17669 server_base.cc:1034] running on GCE node
I20240501 20:08:26.330407 17669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:26.331395 17669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:26.332634 17669 hybrid_clock.cc:648] HybridClock initialized: now 1714594106332584 us; error 44 us; skew 500 ppm
I20240501 20:08:26.333005 17669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:26.339812 17669 webserver.cc:460] Webserver started at http://127.13.251.132:35051/ using document root <none> and password file <none>
I20240501 20:08:26.340754 17669 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:26.340901 17669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:26.346593 17669 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 20:08:26.349726 17683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.351337 17669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:26.351522 17669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "10d2f39dd2434b8e9b9d683defc79425"
format_stamp: "Formatted at 2024-05-01 20:08:13 on dist-test-slave-qswb"
I20240501 20:08:26.352486 17669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:26.373704 17669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:26.374588 17669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:26.374914 17669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:26.375924 17669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:26.377555 17669 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:26.377645 17669 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.377785 17669 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:26.377844 17669 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.428058 17669 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.132:43563
I20240501 20:08:26.428081 17795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.132:43563 every 8 connection(s)
I20240501 20:08:26.429687 17669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:26.432061 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 17669
I20240501 20:08:26.432768 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.133:41735
--local_ip_for_outbound_sockets=127.13.251.133
--tserver_master_addrs=127.13.251.190:43847
--webserver_port=33305
--webserver_interface=127.13.251.133
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:26.446686 17796 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:26.447029 17796 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:26.448599 17796 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:26.451227 16221 ts_manager.cc:194] Re-registered known tserver with Master: 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132:43563)
I20240501 20:08:26.452584 16221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.132:60455
W20240501 20:08:26.785617 17799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:26.785945 17799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:26.786041 17799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:26.793810 17799 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:26.793964 17799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:26.794137 17799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.133
I20240501 20:08:26.803726 17799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.133:41735
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.13.251.133
--webserver_port=33305
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.133
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:26.805137 17799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:26.807026 17799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:26.819904 17805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:26.820497 17807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:26.821383 17804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:26.822314 17799 server_base.cc:1034] running on GCE node
I20240501 20:08:26.822993 17799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:26.824103 17799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:26.825322 17799 hybrid_clock.cc:648] HybridClock initialized: now 1714594106825259 us; error 57 us; skew 500 ppm
I20240501 20:08:26.825810 17799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:26.828539 17799 webserver.cc:460] Webserver started at http://127.13.251.133:33305/ using document root <none> and password file <none>
I20240501 20:08:26.829330 17799 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:26.829465 17799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:26.835898 17799 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20240501 20:08:26.839697 17813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.841367 17799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 20:08:26.841559 17799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cc976bfc5fd643c8bb631f097245c65f"
format_stamp: "Formatted at 2024-05-01 20:08:14 on dist-test-slave-qswb"
I20240501 20:08:26.842621 17799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:26.861078 17799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:26.861974 17799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:26.862274 17799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:26.863312 17799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:26.865028 17799 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:26.865113 17799 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.865226 17799 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:26.865275 17799 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:26.900684 17799 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.133:41735
I20240501 20:08:26.900733 17925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.133:41735 every 8 connection(s)
I20240501 20:08:26.902354 17799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:26.907922 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 17799
I20240501 20:08:26.915532 17926 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:26.915867 17926 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:26.916603 17926 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:26.917930 16221 ts_manager.cc:194] Re-registered known tserver with Master: cc976bfc5fd643c8bb631f097245c65f (127.13.251.133:41735)
I20240501 20:08:26.918704 16221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.133:49827
I20240501 20:08:26.950774 17665 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
I20240501 20:08:26.966993 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.134:40765
--local_ip_for_outbound_sockets=127.13.251.134
--tserver_master_addrs=127.13.251.190:43847
--webserver_port=43885
--webserver_interface=127.13.251.134
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:08:27.303498 17929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:27.303860 17929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:27.304011 17929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:27.311720 17929 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:27.311872 17929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:27.312072 17929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.134
I20240501 20:08:27.321362 17929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.134:40765
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.13.251.134
--webserver_port=43885
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.134
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:27.322808 17929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:27.324714 17929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:27.338156 17937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:27.338793 17935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:27.339381 17934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:27.340252 17929 server_base.cc:1034] running on GCE node
I20240501 20:08:27.340960 17929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:27.342219 17929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:27.343494 17929 hybrid_clock.cc:648] HybridClock initialized: now 1714594107343435 us; error 47 us; skew 500 ppm
I20240501 20:08:27.343907 17929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:27.346290 17929 webserver.cc:460] Webserver started at http://127.13.251.134:43885/ using document root <none> and password file <none>
I20240501 20:08:27.347091 17929 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:27.347216 17929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:27.352844 17929 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:08:27.356053 17943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:27.357712 17929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 20:08:27.357880 17929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "474b2d91e93f46be855e2bf1243bf0c8"
format_stamp: "Formatted at 2024-05-01 20:08:14 on dist-test-slave-qswb"
I20240501 20:08:27.358856 17929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:27.373227 17929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:27.374070 17929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:27.374363 17929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:27.375321 17929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:27.377292 17950 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 20:08:27.382638 17929 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 20:08:27.382759 17929 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 20:08:27.382913 17929 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 20:08:27.386098 17929 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 20:08:27.386229 17929 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:27.386549 17950 tablet_bootstrap.cc:492] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Bootstrap starting.
I20240501 20:08:27.405526 17950 log.cc:824] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:27.410488 17950 tablet_bootstrap.cc:492] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:27.411322 17950 tablet_bootstrap.cc:492] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Bootstrap complete.
I20240501 20:08:27.412879 17950 ts_tablet_manager.cc:1367] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.009s
I20240501 20:08:27.420951 17950 raft_consensus.cc:348] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:27.422255 17950 raft_consensus.cc:729] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474b2d91e93f46be855e2bf1243bf0c8, State: Initialized, Role: FOLLOWER
I20240501 20:08:27.423377 17950 consensus_queue.cc:260] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:27.423619 17950 raft_consensus.cc:388] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:27.423771 17950 raft_consensus.cc:482] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:27.423975 17950 raft_consensus.cc:3037] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:27.429895 17950 raft_consensus.cc:504] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:27.430451 17950 leader_election.cc:302] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 474b2d91e93f46be855e2bf1243bf0c8; no voters:
I20240501 20:08:27.431898 17950 leader_election.cc:288] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 20:08:27.432122 18050 raft_consensus.cc:2781] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:27.437023 18050 raft_consensus.cc:686] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [term 2 LEADER]: Becoming Leader. State: Replica: 474b2d91e93f46be855e2bf1243bf0c8, State: Running, Role: LEADER
I20240501 20:08:27.437832 18050 consensus_queue.cc:237] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:27.440420 17950 ts_tablet_manager.cc:1398] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8: Time spent starting tablet: real 0.027s user 0.020s sys 0.008s
I20240501 20:08:27.443845 17929 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.134:40765
I20240501 20:08:27.443883 18062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.134:40765 every 8 connection(s)
I20240501 20:08:27.445145 17929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:08:27.446136 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 17929
I20240501 20:08:27.447023 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.135:34297
--local_ip_for_outbound_sockets=127.13.251.135
--tserver_master_addrs=127.13.251.190:43847
--webserver_port=38659
--webserver_interface=127.13.251.135
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:27.455322 17796 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
I20240501 20:08:27.462491 18063 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:27.462843 18063 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:27.463910 18063 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:27.466233 16221 ts_manager.cc:194] Re-registered known tserver with Master: 474b2d91e93f46be855e2bf1243bf0c8 (127.13.251.134:40765)
I20240501 20:08:27.466629 16221 catalog_manager.cc:5484] T 09d70c138c8240b394fa6bdb0d3018d9 P 474b2d91e93f46be855e2bf1243bf0c8 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "474b2d91e93f46be855e2bf1243bf0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } } }
I20240501 20:08:27.471812 16221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.134:38605
I20240501 20:08:27.474627 18063 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
W20240501 20:08:27.812958 18067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:27.813266 18067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:27.813392 18067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:27.821499 18067 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:27.821633 18067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:27.821820 18067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.135
I20240501 20:08:27.832175 18067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.135:34297
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.13.251.135
--webserver_port=38659
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.135
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:27.833554 18067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:27.835383 18067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:27.849576 18076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:27.849841 18077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:27.850644 18079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:27.851613 18067 server_base.cc:1034] running on GCE node
I20240501 20:08:27.852353 18067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:27.853416 18067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:27.854655 18067 hybrid_clock.cc:648] HybridClock initialized: now 1714594107854607 us; error 47 us; skew 500 ppm
I20240501 20:08:27.855069 18067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:27.857808 18067 webserver.cc:460] Webserver started at http://127.13.251.135:38659/ using document root <none> and password file <none>
I20240501 20:08:27.858906 18067 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:27.859067 18067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:27.865494 18067 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 20:08:27.869187 18085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:27.870640 18067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:27.870834 18067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "3c7f7a1988b749d6a2bfa4d374e87e69"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-qswb"
I20240501 20:08:27.871820 18067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:27.888098 18067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:27.889008 18067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:27.889343 18067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:27.890383 18067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:27.892910 18067 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:27.893009 18067 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240501 20:08:27.893155 18067 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:27.893213 18067 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:27.921244 17926 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
I20240501 20:08:27.935652 18067 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.135:34297
I20240501 20:08:27.935693 18197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.135:34297 every 8 connection(s)
I20240501 20:08:27.937180 18067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:08:27.939997 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 18067
I20240501 20:08:27.955492 18198 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:27.955849 18198 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:27.956678 18198 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:27.958030 16221 ts_manager.cc:194] Re-registered known tserver with Master: 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135:34297)
I20240501 20:08:27.958799 16221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.135:33123
I20240501 20:08:28.507503 17731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:28.529906 16481 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:28.531896 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:28.532197 18133 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:28.534682 16350 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:08:28.556572 17600 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:28.570319 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:28.777509 17731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:28.779234 18133 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:28.795686 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:28.798822 16481 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:28.817855 17600 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:28.826798 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:28.833061 16350 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:08:28.941705 16370 consensus_queue.cc:237] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
W20240501 20:08:28.952893 16306 consensus_peers.cc:485] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc -> Peer cc976bfc5fd643c8bb631f097245c65f (127.13.251.133:41735): Couldn't send request to peer cc976bfc5fd643c8bb631f097245c65f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9e2c42ca67f4fea9e8dc724658c2090. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:28.961557 18198 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
I20240501 20:08:29.523226 18301 ts_tablet_manager.cc:902] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: Initiating tablet copy from peer b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:29.525537 18301 tablet_copy_client.cc:308] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:44281
I20240501 20:08:29.533156 16390 tablet_copy_service.cc:140] P b75e08cafe834ce580d6666f3a22bdfc: Received BeginTabletCopySession request for tablet c9e2c42ca67f4fea9e8dc724658c2090 from peer cc976bfc5fd643c8bb631f097245c65f ({username='slave'} at 127.13.251.133:40275)
I20240501 20:08:29.533406 16390 tablet_copy_service.cc:161] P b75e08cafe834ce580d6666f3a22bdfc: Beginning new tablet copy session on tablet c9e2c42ca67f4fea9e8dc724658c2090 from peer cc976bfc5fd643c8bb631f097245c65f at {username='slave'} at 127.13.251.133:40275: session id = cc976bfc5fd643c8bb631f097245c65f-c9e2c42ca67f4fea9e8dc724658c2090
I20240501 20:08:29.536135 16390 tablet_copy_source_session.cc:205] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:29.541364 18301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e2c42ca67f4fea9e8dc724658c2090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:29.553853 18301 tablet_copy_client.cc:783] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:29.554220 18301 tablet_copy_client.cc:647] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:29.556891 18301 tablet_copy_client.cc:523] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:29.560515 18301 tablet_bootstrap.cc:492] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: Bootstrap starting.
I20240501 20:08:29.570556 18301 log.cc:824] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:29.603299 18301 tablet_bootstrap.cc:492] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:29.603875 18301 tablet_bootstrap.cc:492] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: Bootstrap complete.
I20240501 20:08:29.604290 18301 ts_tablet_manager.cc:1367] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: Time spent bootstrapping tablet: real 0.044s user 0.036s sys 0.008s
I20240501 20:08:29.608886 18301 raft_consensus.cc:348] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:29.609850 18301 raft_consensus.cc:729] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc976bfc5fd643c8bb631f097245c65f, State: Initialized, Role: FOLLOWER
I20240501 20:08:29.610273 18301 consensus_queue.cc:260] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:29.611886 18301 ts_tablet_manager.cc:1398] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:08:29.613003 16390 tablet_copy_service.cc:342] P b75e08cafe834ce580d6666f3a22bdfc: Request end of tablet copy session cc976bfc5fd643c8bb631f097245c65f-c9e2c42ca67f4fea9e8dc724658c2090 received from {username='slave'} at 127.13.251.133:40275
I20240501 20:08:29.613162 16390 tablet_copy_service.cc:433] P b75e08cafe834ce580d6666f3a22bdfc: ending tablet copy session cc976bfc5fd643c8bb631f097245c65f-c9e2c42ca67f4fea9e8dc724658c2090 on tablet c9e2c42ca67f4fea9e8dc724658c2090 with peer cc976bfc5fd643c8bb631f097245c65f
I20240501 20:08:29.898669 17881 raft_consensus.cc:1203] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:29.900765 18310 raft_consensus.cc:2932] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cc976bfc5fd643c8bb631f097245c65f (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:29.901716 17881 raft_consensus.cc:2932] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cc976bfc5fd643c8bb631f097245c65f (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:29.907711 16224 catalog_manager.cc:5484] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: config changed from index -1 to 4, VOTER cc976bfc5fd643c8bb631f097245c65f (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:29.946277 16501 consensus_queue.cc:237] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
W20240501 20:08:29.956604 16434 consensus_peers.cc:485] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 -> Peer 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131:45387): Couldn't send request to peer 6427cd62d4294811a5b4fd1cbd81d448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eda61e86d34d4ee687006c3cae7a51bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:30.355504 18337 ts_tablet_manager.cc:902] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: Initiating tablet copy from peer 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:30.357699 18337 tablet_copy_client.cc:308] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:41035
I20240501 20:08:30.368563 16521 tablet_copy_service.cc:140] P 63a2e807217a4182a7f7e5a1a105b376: Received BeginTabletCopySession request for tablet eda61e86d34d4ee687006c3cae7a51bf from peer 6427cd62d4294811a5b4fd1cbd81d448 ({username='slave'} at 127.13.251.131:54863)
I20240501 20:08:30.368821 16521 tablet_copy_service.cc:161] P 63a2e807217a4182a7f7e5a1a105b376: Beginning new tablet copy session on tablet eda61e86d34d4ee687006c3cae7a51bf from peer 6427cd62d4294811a5b4fd1cbd81d448 at {username='slave'} at 127.13.251.131:54863: session id = 6427cd62d4294811a5b4fd1cbd81d448-eda61e86d34d4ee687006c3cae7a51bf
I20240501 20:08:30.371683 16521 tablet_copy_source_session.cc:205] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:30.376425 18337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda61e86d34d4ee687006c3cae7a51bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:30.387784 18337 tablet_copy_client.cc:783] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:30.388101 18337 tablet_copy_client.cc:647] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:30.391012 18337 tablet_copy_client.cc:523] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:30.394486 18337 tablet_bootstrap.cc:492] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: Bootstrap starting.
I20240501 20:08:30.405035 18337 log.cc:824] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:30.437095 18337 tablet_bootstrap.cc:492] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:30.437670 18337 tablet_bootstrap.cc:492] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: Bootstrap complete.
I20240501 20:08:30.438143 18337 ts_tablet_manager.cc:1367] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: Time spent bootstrapping tablet: real 0.044s user 0.038s sys 0.005s
I20240501 20:08:30.442840 18337 raft_consensus.cc:348] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:30.443795 18337 raft_consensus.cc:729] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6427cd62d4294811a5b4fd1cbd81d448, State: Initialized, Role: FOLLOWER
I20240501 20:08:30.444239 18337 consensus_queue.cc:260] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:30.445827 18337 ts_tablet_manager.cc:1398] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240501 20:08:30.446870 16521 tablet_copy_service.cc:342] P 63a2e807217a4182a7f7e5a1a105b376: Request end of tablet copy session 6427cd62d4294811a5b4fd1cbd81d448-eda61e86d34d4ee687006c3cae7a51bf received from {username='slave'} at 127.13.251.131:54863
I20240501 20:08:30.447113 16521 tablet_copy_service.cc:433] P 63a2e807217a4182a7f7e5a1a105b376: ending tablet copy session 6427cd62d4294811a5b4fd1cbd81d448-eda61e86d34d4ee687006c3cae7a51bf on tablet eda61e86d34d4ee687006c3cae7a51bf with peer 6427cd62d4294811a5b4fd1cbd81d448
I20240501 20:08:30.784345 17620 raft_consensus.cc:1203] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:30.786324 18345 raft_consensus.cc:2932] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:30.787292 17620 raft_consensus.cc:2932] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:30.795770 16224 catalog_manager.cc:5484] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: config changed from index -1 to 4, VOTER 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:30.854717 16370 consensus_queue.cc:237] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
W20240501 20:08:30.863521 16306 consensus_peers.cc:485] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc -> Peer 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132:43563): Couldn't send request to peer 10d2f39dd2434b8e9b9d683defc79425. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 609385e741e04820bb903f2664108ff2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:31.310439 18370 ts_tablet_manager.cc:902] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: Initiating tablet copy from peer b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:31.312440 18370 tablet_copy_client.cc:308] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:44281
I20240501 20:08:31.319011 16390 tablet_copy_service.cc:140] P b75e08cafe834ce580d6666f3a22bdfc: Received BeginTabletCopySession request for tablet 609385e741e04820bb903f2664108ff2 from peer 10d2f39dd2434b8e9b9d683defc79425 ({username='slave'} at 127.13.251.132:56559)
I20240501 20:08:31.319197 16390 tablet_copy_service.cc:161] P b75e08cafe834ce580d6666f3a22bdfc: Beginning new tablet copy session on tablet 609385e741e04820bb903f2664108ff2 from peer 10d2f39dd2434b8e9b9d683defc79425 at {username='slave'} at 127.13.251.132:56559: session id = 10d2f39dd2434b8e9b9d683defc79425-609385e741e04820bb903f2664108ff2
I20240501 20:08:31.321712 16390 tablet_copy_source_session.cc:205] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:31.326437 18370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609385e741e04820bb903f2664108ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:31.337415 18370 tablet_copy_client.cc:783] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:31.337723 18370 tablet_copy_client.cc:647] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:31.340024 18370 tablet_copy_client.cc:523] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:31.343405 18370 tablet_bootstrap.cc:492] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: Bootstrap starting.
I20240501 20:08:31.353173 18370 log.cc:824] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:31.390221 18370 tablet_bootstrap.cc:492] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:31.390794 18370 tablet_bootstrap.cc:492] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: Bootstrap complete.
I20240501 20:08:31.391229 18370 ts_tablet_manager.cc:1367] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: Time spent bootstrapping tablet: real 0.048s user 0.043s sys 0.004s
I20240501 20:08:31.395990 18370 raft_consensus.cc:348] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:31.396827 18370 raft_consensus.cc:729] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d2f39dd2434b8e9b9d683defc79425, State: Initialized, Role: FOLLOWER
I20240501 20:08:31.397267 18370 consensus_queue.cc:260] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:31.398691 18370 ts_tablet_manager.cc:1398] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425: Time spent starting tablet: real 0.007s user 0.010s sys 0.000s
I20240501 20:08:31.399719 16390 tablet_copy_service.cc:342] P b75e08cafe834ce580d6666f3a22bdfc: Request end of tablet copy session 10d2f39dd2434b8e9b9d683defc79425-609385e741e04820bb903f2664108ff2 received from {username='slave'} at 127.13.251.132:56559
I20240501 20:08:31.399842 16390 tablet_copy_service.cc:433] P b75e08cafe834ce580d6666f3a22bdfc: ending tablet copy session 10d2f39dd2434b8e9b9d683defc79425-609385e741e04820bb903f2664108ff2 on tablet 609385e741e04820bb903f2664108ff2 with peer 10d2f39dd2434b8e9b9d683defc79425
I20240501 20:08:31.914531 17751 raft_consensus.cc:1203] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:31.916442 18381 raft_consensus.cc:2932] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:31.917287 17751 raft_consensus.cc:2932] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:31.919271 18381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40040 (request call id 0) took 1065 ms. Trace:
I20240501 20:08:31.919500 18381 rpcz_store.cc:276] 0501 20:08:30.854159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:08:30.854229 (+ 70us) service_pool.cc:225] Handling call
0501 20:08:30.855461 (+ 1232us) log.cc:842] Serialized 263 byte log entry
0501 20:08:31.919217 (+1063756us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":79,"raft.queue_time_us":48,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":265,"spinlock_wait_cycles":13184,"thread_start_us":241,"threads_started":1,"wal-append.queue_time_us":372,"wal-append.run_cpu_time_us":563,"wal-append.run_wall_time_us":1000645}
I20240501 20:08:31.922744 16221 catalog_manager.cc:5484] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: config changed from index -1 to 4, VOTER 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:31.966109 16501 consensus_queue.cc:237] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
W20240501 20:08:31.978693 16434 consensus_peers.cc:485] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 -> Peer 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135:34297): Couldn't send request to peer 3c7f7a1988b749d6a2bfa4d374e87e69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb81d991e8a3455db9f94b6ef07b86ee. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:32.432312 18408 ts_tablet_manager.cc:902] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: Initiating tablet copy from peer 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:32.436097 18408 tablet_copy_client.cc:308] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:41035
I20240501 20:08:32.443392 16521 tablet_copy_service.cc:140] P 63a2e807217a4182a7f7e5a1a105b376: Received BeginTabletCopySession request for tablet fb81d991e8a3455db9f94b6ef07b86ee from peer 3c7f7a1988b749d6a2bfa4d374e87e69 ({username='slave'} at 127.13.251.135:50253)
I20240501 20:08:32.443631 16521 tablet_copy_service.cc:161] P 63a2e807217a4182a7f7e5a1a105b376: Beginning new tablet copy session on tablet fb81d991e8a3455db9f94b6ef07b86ee from peer 3c7f7a1988b749d6a2bfa4d374e87e69 at {username='slave'} at 127.13.251.135:50253: session id = 3c7f7a1988b749d6a2bfa4d374e87e69-fb81d991e8a3455db9f94b6ef07b86ee
I20240501 20:08:32.446700 16521 tablet_copy_source_session.cc:205] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:32.452344 18408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb81d991e8a3455db9f94b6ef07b86ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:32.463135 18408 tablet_copy_client.cc:783] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:32.463501 18408 tablet_copy_client.cc:647] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:32.465807 18408 tablet_copy_client.cc:523] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:32.469094 18408 tablet_bootstrap.cc:492] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: Bootstrap starting.
I20240501 20:08:32.479365 18408 log.cc:824] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:32.512940 18408 tablet_bootstrap.cc:492] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:32.513490 18408 tablet_bootstrap.cc:492] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: Bootstrap complete.
I20240501 20:08:32.513902 18408 ts_tablet_manager.cc:1367] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: Time spent bootstrapping tablet: real 0.045s user 0.036s sys 0.008s
I20240501 20:08:32.518376 18408 raft_consensus.cc:348] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:32.519263 18408 raft_consensus.cc:729] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7f7a1988b749d6a2bfa4d374e87e69, State: Initialized, Role: FOLLOWER
I20240501 20:08:32.519706 18408 consensus_queue.cc:260] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:32.521318 18408 ts_tablet_manager.cc:1398] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20240501 20:08:32.522482 16521 tablet_copy_service.cc:342] P 63a2e807217a4182a7f7e5a1a105b376: Request end of tablet copy session 3c7f7a1988b749d6a2bfa4d374e87e69-fb81d991e8a3455db9f94b6ef07b86ee received from {username='slave'} at 127.13.251.135:50253
I20240501 20:08:32.522758 16521 tablet_copy_service.cc:433] P 63a2e807217a4182a7f7e5a1a105b376: ending tablet copy session 3c7f7a1988b749d6a2bfa4d374e87e69-fb81d991e8a3455db9f94b6ef07b86ee on tablet fb81d991e8a3455db9f94b6ef07b86ee with peer 3c7f7a1988b749d6a2bfa4d374e87e69
I20240501 20:08:33.025590 18153 raft_consensus.cc:1203] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:33.027812 18417 raft_consensus.cc:2932] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:33.028831 18153 raft_consensus.cc:2932] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:33.031919 18417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50404 (request call id 0) took 1066 ms. Trace:
I20240501 20:08:33.032162 18417 rpcz_store.cc:276] 0501 20:08:31.965524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:08:31.965603 (+ 79us) service_pool.cc:225] Handling call
0501 20:08:31.966766 (+ 1163us) log.cc:842] Serialized 263 byte log entry
0501 20:08:33.031730 (+1064964us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":78,"raft.queue_time_us":76,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":181,"thread_start_us":274,"threads_started":1,"wal-append.queue_time_us":393,"wal-append.run_cpu_time_us":547,"wal-append.run_wall_time_us":1000590}
I20240501 20:08:33.037207 16221 catalog_manager.cc:5484] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 reported cstate change: config changed from index -1 to 4, VOTER 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:33.070735 16370 consensus_queue.cc:237] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
W20240501 20:08:33.081205 16303 consensus_peers.cc:485] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc -> Peer 474b2d91e93f46be855e2bf1243bf0c8 (127.13.251.134:40765): Couldn't send request to peer 474b2d91e93f46be855e2bf1243bf0c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 699e4265f7d14e47b55ab17594554f77. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:33.530414 18444 ts_tablet_manager.cc:902] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: Initiating tablet copy from peer b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:33.532506 18444 tablet_copy_client.cc:308] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:44281
I20240501 20:08:33.539216 16390 tablet_copy_service.cc:140] P b75e08cafe834ce580d6666f3a22bdfc: Received BeginTabletCopySession request for tablet 699e4265f7d14e47b55ab17594554f77 from peer 474b2d91e93f46be855e2bf1243bf0c8 ({username='slave'} at 127.13.251.134:34323)
I20240501 20:08:33.539407 16390 tablet_copy_service.cc:161] P b75e08cafe834ce580d6666f3a22bdfc: Beginning new tablet copy session on tablet 699e4265f7d14e47b55ab17594554f77 from peer 474b2d91e93f46be855e2bf1243bf0c8 at {username='slave'} at 127.13.251.134:34323: session id = 474b2d91e93f46be855e2bf1243bf0c8-699e4265f7d14e47b55ab17594554f77
I20240501 20:08:33.541622 16390 tablet_copy_source_session.cc:205] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:33.544390 18444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699e4265f7d14e47b55ab17594554f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.551940 18444 tablet_copy_client.cc:783] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:33.552335 18444 tablet_copy_client.cc:647] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:33.554898 18444 tablet_copy_client.cc:523] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:33.557940 18444 tablet_bootstrap.cc:492] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: Bootstrap starting.
I20240501 20:08:33.599947 18444 tablet_bootstrap.cc:492] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:33.600478 18444 tablet_bootstrap.cc:492] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: Bootstrap complete.
I20240501 20:08:33.600764 18444 ts_tablet_manager.cc:1367] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: Time spent bootstrapping tablet: real 0.043s user 0.037s sys 0.006s
I20240501 20:08:33.602159 18444 raft_consensus.cc:348] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:33.602659 18444 raft_consensus.cc:729] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474b2d91e93f46be855e2bf1243bf0c8, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.602821 18444 consensus_queue.cc:260] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:33.603511 18444 ts_tablet_manager.cc:1398] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:33.604665 16390 tablet_copy_service.cc:342] P b75e08cafe834ce580d6666f3a22bdfc: Request end of tablet copy session 474b2d91e93f46be855e2bf1243bf0c8-699e4265f7d14e47b55ab17594554f77 received from {username='slave'} at 127.13.251.134:34323
I20240501 20:08:33.604852 16390 tablet_copy_service.cc:433] P b75e08cafe834ce580d6666f3a22bdfc: ending tablet copy session 474b2d91e93f46be855e2bf1243bf0c8-699e4265f7d14e47b55ab17594554f77 on tablet 699e4265f7d14e47b55ab17594554f77 with peer 474b2d91e93f46be855e2bf1243bf0c8
I20240501 20:08:34.015121 18013 raft_consensus.cc:1203] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:34.016924 18453 raft_consensus.cc:2932] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 474b2d91e93f46be855e2bf1243bf0c8 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } } }
I20240501 20:08:34.017674 18013 raft_consensus.cc:2932] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 474b2d91e93f46be855e2bf1243bf0c8 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } } }
I20240501 20:08:34.023202 16221 catalog_manager.cc:5484] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: config changed from index -1 to 4, VOTER 474b2d91e93f46be855e2bf1243bf0c8 (127.13.251.134) added. New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } } }
I20240501 20:08:34.059505 16501 consensus_queue.cc:237] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
W20240501 20:08:34.069664 16437 consensus_peers.cc:485] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 -> Peer cc976bfc5fd643c8bb631f097245c65f (127.13.251.133:41735): Couldn't send request to peer cc976bfc5fd643c8bb631f097245c65f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b380bf3c85748cfb4843ae21eadc74b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:34.650194 18479 ts_tablet_manager.cc:902] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: Initiating tablet copy from peer 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:34.651718 18479 tablet_copy_client.cc:308] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:41035
I20240501 20:08:34.658943 16521 tablet_copy_service.cc:140] P 63a2e807217a4182a7f7e5a1a105b376: Received BeginTabletCopySession request for tablet 0b380bf3c85748cfb4843ae21eadc74b from peer cc976bfc5fd643c8bb631f097245c65f ({username='slave'} at 127.13.251.133:41085)
I20240501 20:08:34.659140 16521 tablet_copy_service.cc:161] P 63a2e807217a4182a7f7e5a1a105b376: Beginning new tablet copy session on tablet 0b380bf3c85748cfb4843ae21eadc74b from peer cc976bfc5fd643c8bb631f097245c65f at {username='slave'} at 127.13.251.133:41085: session id = cc976bfc5fd643c8bb631f097245c65f-0b380bf3c85748cfb4843ae21eadc74b
I20240501 20:08:34.661762 16521 tablet_copy_source_session.cc:205] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:34.663229 18479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b380bf3c85748cfb4843ae21eadc74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:34.667831 18479 tablet_copy_client.cc:783] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:34.668056 18479 tablet_copy_client.cc:647] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:34.669689 18479 tablet_copy_client.cc:523] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:34.672040 18479 tablet_bootstrap.cc:492] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: Bootstrap starting.
I20240501 20:08:34.700762 18479 tablet_bootstrap.cc:492] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:34.701207 18479 tablet_bootstrap.cc:492] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: Bootstrap complete.
I20240501 20:08:34.701463 18479 ts_tablet_manager.cc:1367] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.000s
I20240501 20:08:34.702293 18479 raft_consensus.cc:348] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:34.702651 18479 raft_consensus.cc:729] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc976bfc5fd643c8bb631f097245c65f, State: Initialized, Role: FOLLOWER
I20240501 20:08:34.702800 18479 consensus_queue.cc:260] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:34.703457 18479 ts_tablet_manager.cc:1398] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:08:34.704463 16521 tablet_copy_service.cc:342] P 63a2e807217a4182a7f7e5a1a105b376: Request end of tablet copy session cc976bfc5fd643c8bb631f097245c65f-0b380bf3c85748cfb4843ae21eadc74b received from {username='slave'} at 127.13.251.133:41085
I20240501 20:08:34.704591 16521 tablet_copy_service.cc:433] P 63a2e807217a4182a7f7e5a1a105b376: ending tablet copy session cc976bfc5fd643c8bb631f097245c65f-0b380bf3c85748cfb4843ae21eadc74b on tablet 0b380bf3c85748cfb4843ae21eadc74b with peer cc976bfc5fd643c8bb631f097245c65f
I20240501 20:08:35.256275 17881 raft_consensus.cc:1203] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:35.257826 18489 raft_consensus.cc:2932] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cc976bfc5fd643c8bb631f097245c65f (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:35.258450 17881 raft_consensus.cc:2932] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cc976bfc5fd643c8bb631f097245c65f (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:35.260954 18489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50414 (request call id 0) took 1202 ms. Trace:
I20240501 20:08:35.261126 18489 rpcz_store.cc:276] 0501 20:08:34.058924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:08:34.058999 (+ 75us) service_pool.cc:225] Handling call
0501 20:08:34.060276 (+ 1277us) log.cc:842] Serialized 263 byte log entry
0501 20:08:35.260897 (+1200621us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":69,"raft.queue_time_us":62,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":250,"thread_start_us":233,"threads_started":1,"wal-append.queue_time_us":325,"wal-append.run_cpu_time_us":551,"wal-append.run_wall_time_us":1000554}
I20240501 20:08:35.264050 16228 catalog_manager.cc:5484] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f reported cstate change: config changed from index -1 to 4, VOTER cc976bfc5fd643c8bb631f097245c65f (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:35.296332 16370 consensus_queue.cc:237] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
W20240501 20:08:35.305222 16303 consensus_peers.cc:485] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc -> Peer 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131:45387): Couldn't send request to peer 6427cd62d4294811a5b4fd1cbd81d448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3acfb78e3127445db565fc5ceaca8b9e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:35.842761 18516 ts_tablet_manager.cc:902] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: Initiating tablet copy from peer b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:35.844377 18516 tablet_copy_client.cc:308] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:44281
I20240501 20:08:35.852174 16390 tablet_copy_service.cc:140] P b75e08cafe834ce580d6666f3a22bdfc: Received BeginTabletCopySession request for tablet 3acfb78e3127445db565fc5ceaca8b9e from peer 6427cd62d4294811a5b4fd1cbd81d448 ({username='slave'} at 127.13.251.131:53649)
I20240501 20:08:35.852396 16390 tablet_copy_service.cc:161] P b75e08cafe834ce580d6666f3a22bdfc: Beginning new tablet copy session on tablet 3acfb78e3127445db565fc5ceaca8b9e from peer 6427cd62d4294811a5b4fd1cbd81d448 at {username='slave'} at 127.13.251.131:53649: session id = 6427cd62d4294811a5b4fd1cbd81d448-3acfb78e3127445db565fc5ceaca8b9e
I20240501 20:08:35.854598 16390 tablet_copy_source_session.cc:205] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:35.856066 18516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acfb78e3127445db565fc5ceaca8b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.861457 18516 tablet_copy_client.cc:783] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:35.861755 18516 tablet_copy_client.cc:647] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:35.864029 18516 tablet_copy_client.cc:523] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:35.866832 18516 tablet_bootstrap.cc:492] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: Bootstrap starting.
I20240501 20:08:35.903602 18516 tablet_bootstrap.cc:492] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:35.904171 18516 tablet_bootstrap.cc:492] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: Bootstrap complete.
I20240501 20:08:35.904459 18516 ts_tablet_manager.cc:1367] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.013s
I20240501 20:08:35.905360 18516 raft_consensus.cc:348] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:35.905808 18516 raft_consensus.cc:729] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6427cd62d4294811a5b4fd1cbd81d448, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.906056 18516 consensus_queue.cc:260] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:35.907003 18516 ts_tablet_manager.cc:1398] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:08:35.908013 16390 tablet_copy_service.cc:342] P b75e08cafe834ce580d6666f3a22bdfc: Request end of tablet copy session 6427cd62d4294811a5b4fd1cbd81d448-3acfb78e3127445db565fc5ceaca8b9e received from {username='slave'} at 127.13.251.131:53649
I20240501 20:08:35.908237 16390 tablet_copy_service.cc:433] P b75e08cafe834ce580d6666f3a22bdfc: ending tablet copy session 6427cd62d4294811a5b4fd1cbd81d448-3acfb78e3127445db565fc5ceaca8b9e on tablet 3acfb78e3127445db565fc5ceaca8b9e with peer 6427cd62d4294811a5b4fd1cbd81d448
I20240501 20:08:36.457531 17620 raft_consensus.cc:1203] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:36.459669 18526 raft_consensus.cc:2932] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:36.461408 17620 raft_consensus.cc:2932] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:36.464524 18526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40060 (request call id 0) took 1168 ms. Trace:
I20240501 20:08:36.464780 18526 rpcz_store.cc:276] 0501 20:08:35.295724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:08:35.295798 (+ 74us) service_pool.cc:225] Handling call
0501 20:08:35.297020 (+ 1222us) log.cc:842] Serialized 263 byte log entry
0501 20:08:36.464445 (+1167425us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":65,"raft.queue_time_us":57,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":208,"thread_start_us":234,"threads_started":1,"wal-append.queue_time_us":318,"wal-append.run_cpu_time_us":445,"wal-append.run_wall_time_us":1000550}
I20240501 20:08:36.469331 16228 catalog_manager.cc:5484] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: config changed from index -1 to 4, VOTER 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:36.510329 16501 consensus_queue.cc:237] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
W20240501 20:08:36.518921 16437 consensus_peers.cc:485] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 -> Peer 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132:43563): Couldn't send request to peer 10d2f39dd2434b8e9b9d683defc79425. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 608ac70c00014df693e8bd48925af0c3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:37.056356 18553 ts_tablet_manager.cc:902] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: Initiating tablet copy from peer 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:37.057742 18553 tablet_copy_client.cc:308] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:41035
I20240501 20:08:37.065349 16521 tablet_copy_service.cc:140] P 63a2e807217a4182a7f7e5a1a105b376: Received BeginTabletCopySession request for tablet 608ac70c00014df693e8bd48925af0c3 from peer 10d2f39dd2434b8e9b9d683defc79425 ({username='slave'} at 127.13.251.132:35725)
I20240501 20:08:37.065536 16521 tablet_copy_service.cc:161] P 63a2e807217a4182a7f7e5a1a105b376: Beginning new tablet copy session on tablet 608ac70c00014df693e8bd48925af0c3 from peer 10d2f39dd2434b8e9b9d683defc79425 at {username='slave'} at 127.13.251.132:35725: session id = 10d2f39dd2434b8e9b9d683defc79425-608ac70c00014df693e8bd48925af0c3
I20240501 20:08:37.067893 16521 tablet_copy_source_session.cc:205] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:37.069362 18553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608ac70c00014df693e8bd48925af0c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:37.074920 18553 tablet_copy_client.cc:783] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:37.075234 18553 tablet_copy_client.cc:647] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:37.077425 18553 tablet_copy_client.cc:523] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:37.080404 18553 tablet_bootstrap.cc:492] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: Bootstrap starting.
I20240501 20:08:37.116132 18553 tablet_bootstrap.cc:492] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:37.116595 18553 tablet_bootstrap.cc:492] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: Bootstrap complete.
I20240501 20:08:37.116919 18553 ts_tablet_manager.cc:1367] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.007s
I20240501 20:08:37.117878 18553 raft_consensus.cc:348] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:37.118291 18553 raft_consensus.cc:729] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d2f39dd2434b8e9b9d683defc79425, State: Initialized, Role: FOLLOWER
I20240501 20:08:37.118499 18553 consensus_queue.cc:260] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:37.119449 18553 ts_tablet_manager.cc:1398] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240501 20:08:37.120460 16521 tablet_copy_service.cc:342] P 63a2e807217a4182a7f7e5a1a105b376: Request end of tablet copy session 10d2f39dd2434b8e9b9d683defc79425-608ac70c00014df693e8bd48925af0c3 received from {username='slave'} at 127.13.251.132:35725
I20240501 20:08:37.120585 16521 tablet_copy_service.cc:433] P 63a2e807217a4182a7f7e5a1a105b376: ending tablet copy session 10d2f39dd2434b8e9b9d683defc79425-608ac70c00014df693e8bd48925af0c3 on tablet 608ac70c00014df693e8bd48925af0c3 with peer 10d2f39dd2434b8e9b9d683defc79425
I20240501 20:08:37.577647 17751 raft_consensus.cc:1203] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:37.579025 18562 raft_consensus.cc:2932] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:37.580442 17751 raft_consensus.cc:2932] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:37.582139 18562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:45812 (request call id 0) took 1072 ms. Trace:
I20240501 20:08:37.582288 18562 rpcz_store.cc:276] 0501 20:08:36.509688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:08:36.509771 (+ 83us) service_pool.cc:225] Handling call
0501 20:08:36.511041 (+ 1270us) log.cc:842] Serialized 263 byte log entry
0501 20:08:37.582070 (+1071029us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":91,"raft.queue_time_us":56,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":221,"thread_start_us":250,"threads_started":1,"wal-append.queue_time_us":356,"wal-append.run_cpu_time_us":650,"wal-append.run_wall_time_us":1000742}
I20240501 20:08:37.585040 16228 catalog_manager.cc:5484] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: config changed from index -1 to 4, VOTER 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:37.619930 16370 consensus_queue.cc:237] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
W20240501 20:08:37.628984 16303 consensus_peers.cc:485] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc -> Peer 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135:34297): Couldn't send request to peer 3c7f7a1988b749d6a2bfa4d374e87e69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f522ede49dac4f7193158c6bca98ed26. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:38.042801 18589 ts_tablet_manager.cc:902] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: Initiating tablet copy from peer b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:38.044468 18589 tablet_copy_client.cc:308] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:44281
I20240501 20:08:38.051600 16390 tablet_copy_service.cc:140] P b75e08cafe834ce580d6666f3a22bdfc: Received BeginTabletCopySession request for tablet f522ede49dac4f7193158c6bca98ed26 from peer 3c7f7a1988b749d6a2bfa4d374e87e69 ({username='slave'} at 127.13.251.135:58547)
I20240501 20:08:38.051838 16390 tablet_copy_service.cc:161] P b75e08cafe834ce580d6666f3a22bdfc: Beginning new tablet copy session on tablet f522ede49dac4f7193158c6bca98ed26 from peer 3c7f7a1988b749d6a2bfa4d374e87e69 at {username='slave'} at 127.13.251.135:58547: session id = 3c7f7a1988b749d6a2bfa4d374e87e69-f522ede49dac4f7193158c6bca98ed26
I20240501 20:08:38.054447 16390 tablet_copy_source_session.cc:205] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:38.055981 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f522ede49dac4f7193158c6bca98ed26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:38.061365 18589 tablet_copy_client.cc:783] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:38.061666 18589 tablet_copy_client.cc:647] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:38.063653 18589 tablet_copy_client.cc:523] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:38.066393 18589 tablet_bootstrap.cc:492] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: Bootstrap starting.
I20240501 20:08:38.101696 18589 tablet_bootstrap.cc:492] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:38.102653 18589 tablet_bootstrap.cc:492] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: Bootstrap complete.
I20240501 20:08:38.102947 18589 ts_tablet_manager.cc:1367] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.004s
I20240501 20:08:38.103837 18589 raft_consensus.cc:348] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:38.104234 18589 raft_consensus.cc:729] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7f7a1988b749d6a2bfa4d374e87e69, State: Initialized, Role: FOLLOWER
I20240501 20:08:38.104413 18589 consensus_queue.cc:260] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:38.105147 18589 ts_tablet_manager.cc:1398] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 20:08:38.106079 16390 tablet_copy_service.cc:342] P b75e08cafe834ce580d6666f3a22bdfc: Request end of tablet copy session 3c7f7a1988b749d6a2bfa4d374e87e69-f522ede49dac4f7193158c6bca98ed26 received from {username='slave'} at 127.13.251.135:58547
I20240501 20:08:38.106220 16390 tablet_copy_service.cc:433] P b75e08cafe834ce580d6666f3a22bdfc: ending tablet copy session 3c7f7a1988b749d6a2bfa4d374e87e69-f522ede49dac4f7193158c6bca98ed26 on tablet f522ede49dac4f7193158c6bca98ed26 with peer 3c7f7a1988b749d6a2bfa4d374e87e69
I20240501 20:08:38.421175 18153 raft_consensus.cc:1203] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:38.422770 18597 raft_consensus.cc:2932] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:38.423584 18153 raft_consensus.cc:2932] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:38.428892 16221 catalog_manager.cc:5484] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: config changed from index -1 to 4, VOTER 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:38.548617 17731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:38.563212 18133 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:38.570917 16481 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:38.578171 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:38.594050 16350 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:08:38.613155 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:38.629236 17600 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:38.685490 16370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f522ede49dac4f7193158c6bca98ed26"
dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48426
I20240501 20:08:38.685815 16370 raft_consensus.cc:595] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:38.786754 17731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:38.808446 18133 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:38.809948 16481 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:38.830916 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:38.844759 16350 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:08:38.849738 17600 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:38.860280 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:38.970528 18692 raft_consensus.cc:979] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: : Instructing follower 3c7f7a1988b749d6a2bfa4d374e87e69 to start an election
I20240501 20:08:38.970750 18692 raft_consensus.cc:1067] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Signalling peer 3c7f7a1988b749d6a2bfa4d374e87e69 to start an election
I20240501 20:08:38.971901 18153 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f522ede49dac4f7193158c6bca98ed26"
dest_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69"
from {username='slave'} at 127.13.251.129:58741
I20240501 20:08:38.972244 18153 raft_consensus.cc:482] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:38.972466 18153 raft_consensus.cc:3037] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:38.975854 18153 raft_consensus.cc:504] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:38.977596 18153 leader_election.cc:288] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [CANDIDATE]: Term 2 election: Requested vote from peers b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:38.978356 16370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f522ede49dac4f7193158c6bca98ed26" candidate_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
I20240501 20:08:38.978729 16370 raft_consensus.cc:3032] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:38.978838 16370 raft_consensus.cc:729] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:38.979220 16370 consensus_queue.cc:260] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:38.979804 16370 raft_consensus.cc:3037] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:38.983125 16370 raft_consensus.cc:2454] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c7f7a1988b749d6a2bfa4d374e87e69 in term 2.
I20240501 20:08:38.983831 18086 leader_election.cc:302] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3c7f7a1988b749d6a2bfa4d374e87e69, b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:38.984804 18693 raft_consensus.cc:2781] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:38.985452 18693 raft_consensus.cc:686] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 2 LEADER]: Becoming Leader. State: Replica: 3c7f7a1988b749d6a2bfa4d374e87e69, State: Running, Role: LEADER
I20240501 20:08:38.986078 18693 consensus_queue.cc:237] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:38.990171 16229 catalog_manager.cc:5484] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 reported cstate change: term changed from 1 to 2, leader changed from b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) to 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135). New cstate: current_term: 2 leader_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:39.075407 17731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.077517 16481 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:39.085963 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.093179 16350 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:39.108739 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.109162 17600 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.114238 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.200029 16501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "608ac70c00014df693e8bd48925af0c3"
dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45840
I20240501 20:08:39.200278 16501 raft_consensus.cc:595] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:39.297360 17731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.304786 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.312561 16481 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:39.312579 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.344655 17600 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.345887 16350 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:39.378414 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.534060 17731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.535827 16370 raft_consensus.cc:1261] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Refusing update from remote peer 3c7f7a1988b749d6a2bfa4d374e87e69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:39.537021 18796 consensus_queue.cc:1035] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:39.550761 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.551296 16481 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:39.554031 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.575223 17600 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.579181 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.598416 16350 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:39.655903 18562 raft_consensus.cc:979] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: : Instructing follower 10d2f39dd2434b8e9b9d683defc79425 to start an election
I20240501 20:08:39.656123 18562 raft_consensus.cc:1067] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Signalling peer 10d2f39dd2434b8e9b9d683defc79425 to start an election
I20240501 20:08:39.657194 17751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "608ac70c00014df693e8bd48925af0c3"
dest_uuid: "10d2f39dd2434b8e9b9d683defc79425"
from {username='slave'} at 127.13.251.130:43433
I20240501 20:08:39.657485 17751 raft_consensus.cc:482] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:39.657668 17751 raft_consensus.cc:3037] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:39.659994 17751 raft_consensus.cc:504] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:39.661569 17751 leader_election.cc:288] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [CANDIDATE]: Term 2 election: Requested vote from peers 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:39.662215 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608ac70c00014df693e8bd48925af0c3" candidate_uuid: "10d2f39dd2434b8e9b9d683defc79425" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
I20240501 20:08:39.662499 16501 raft_consensus.cc:3032] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:39.662623 16501 raft_consensus.cc:729] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:39.662969 16501 consensus_queue.cc:260] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:39.663489 16501 raft_consensus.cc:3037] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:39.667863 16501 raft_consensus.cc:2454] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10d2f39dd2434b8e9b9d683defc79425 in term 2.
I20240501 20:08:39.668573 17686 leader_election.cc:302] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 10d2f39dd2434b8e9b9d683defc79425, 63a2e807217a4182a7f7e5a1a105b376; no voters:
I20240501 20:08:39.672523 18803 raft_consensus.cc:2781] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:39.673045 18803 raft_consensus.cc:686] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 2 LEADER]: Becoming Leader. State: Replica: 10d2f39dd2434b8e9b9d683defc79425, State: Running, Role: LEADER
I20240501 20:08:39.673615 18803 consensus_queue.cc:237] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:39.678151 16229 catalog_manager.cc:5484] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 reported cstate change: term changed from 1 to 2, leader changed from 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) to 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132). New cstate: current_term: 2 leader_uuid: "10d2f39dd2434b8e9b9d683defc79425" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:39.684532 16370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3acfb78e3127445db565fc5ceaca8b9e"
dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48494
I20240501 20:08:39.684743 16370 raft_consensus.cc:595] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:39.783071 17731 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.789528 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.796404 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.800349 18846 raft_consensus.cc:979] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: : Instructing follower 6427cd62d4294811a5b4fd1cbd81d448 to start an election
I20240501 20:08:39.800611 18692 raft_consensus.cc:1067] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Signalling peer 6427cd62d4294811a5b4fd1cbd81d448 to start an election
I20240501 20:08:39.801535 17620 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3acfb78e3127445db565fc5ceaca8b9e"
dest_uuid: "6427cd62d4294811a5b4fd1cbd81d448"
from {username='slave'} at 127.13.251.129:43741
I20240501 20:08:39.801872 17620 raft_consensus.cc:482] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:39.802101 17620 raft_consensus.cc:3037] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:39.805203 17620 raft_consensus.cc:504] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:39.807601 16370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acfb78e3127445db565fc5ceaca8b9e" candidate_uuid: "6427cd62d4294811a5b4fd1cbd81d448" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
I20240501 20:08:39.807838 16370 raft_consensus.cc:3032] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:39.807934 16370 raft_consensus.cc:729] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:39.808176 16370 consensus_queue.cc:260] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:39.808387 16370 raft_consensus.cc:3037] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:39.811935 16370 raft_consensus.cc:2454] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6427cd62d4294811a5b4fd1cbd81d448 in term 2.
I20240501 20:08:39.812960 17553 leader_election.cc:302] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6427cd62d4294811a5b4fd1cbd81d448, b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:39.814903 17620 leader_election.cc:288] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [CANDIDATE]: Term 2 election: Requested vote from peers b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:39.815987 16481 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:39.817837 16350 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:39.837546 18847 raft_consensus.cc:2781] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:39.844220 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.849072 18847 raft_consensus.cc:686] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 2 LEADER]: Becoming Leader. State: Replica: 6427cd62d4294811a5b4fd1cbd81d448, State: Running, Role: LEADER
I20240501 20:08:39.850009 18847 consensus_queue.cc:237] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:39.864602 17600 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.883739 16229 catalog_manager.cc:5484] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 reported cstate change: term changed from 1 to 2, leader changed from b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) to 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131). New cstate: current_term: 2 leader_uuid: "6427cd62d4294811a5b4fd1cbd81d448" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:40.046288 17731 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.054868 16481 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:40.059746 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.064529 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.078742 17600 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.087110 16350 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:40.115744 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.172590 16501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eda61e86d34d4ee687006c3cae7a51bf"
dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45892
I20240501 20:08:40.172796 16501 raft_consensus.cc:595] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:40.272269 17731 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.275064 16501 raft_consensus.cc:1261] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Refusing update from remote peer 10d2f39dd2434b8e9b9d683defc79425: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:40.283457 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.284241 18921 consensus_queue.cc:1035] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:40.293671 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.294008 16350 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:40.314930 17600 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.317199 16481 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:40.332912 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.470747 16370 raft_consensus.cc:1261] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Refusing update from remote peer 6427cd62d4294811a5b4fd1cbd81d448: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:40.471472 18940 consensus_queue.cc:1035] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:40.498766 18802 raft_consensus.cc:979] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: : Instructing follower 6427cd62d4294811a5b4fd1cbd81d448 to start an election
I20240501 20:08:40.498927 18802 raft_consensus.cc:1067] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Signalling peer 6427cd62d4294811a5b4fd1cbd81d448 to start an election
I20240501 20:08:40.499584 17620 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eda61e86d34d4ee687006c3cae7a51bf"
dest_uuid: "6427cd62d4294811a5b4fd1cbd81d448"
from {username='slave'} at 127.13.251.130:42913
I20240501 20:08:40.499809 17620 raft_consensus.cc:482] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:40.499922 17620 raft_consensus.cc:3037] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:40.502139 17620 raft_consensus.cc:504] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:40.502810 17620 leader_election.cc:288] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [CANDIDATE]: Term 2 election: Requested vote from peers 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:40.503369 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda61e86d34d4ee687006c3cae7a51bf" candidate_uuid: "6427cd62d4294811a5b4fd1cbd81d448" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
I20240501 20:08:40.503597 16501 raft_consensus.cc:3032] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:40.503679 16501 raft_consensus.cc:729] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:40.503973 16501 consensus_queue.cc:260] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:40.504233 16501 raft_consensus.cc:3037] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:40.506683 16501 raft_consensus.cc:2454] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6427cd62d4294811a5b4fd1cbd81d448 in term 2.
I20240501 20:08:40.507104 17555 leader_election.cc:302] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376, 6427cd62d4294811a5b4fd1cbd81d448; no voters:
I20240501 20:08:40.507414 18942 raft_consensus.cc:2781] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:40.507588 18942 raft_consensus.cc:686] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 2 LEADER]: Becoming Leader. State: Replica: 6427cd62d4294811a5b4fd1cbd81d448, State: Running, Role: LEADER
I20240501 20:08:40.507918 18942 consensus_queue.cc:237] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:40.510427 16221 catalog_manager.cc:5484] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 reported cstate change: term changed from 1 to 2, leader changed from 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) to 6427cd62d4294811a5b4fd1cbd81d448 (127.13.251.131). New cstate: current_term: 2 leader_uuid: "6427cd62d4294811a5b4fd1cbd81d448" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:40.558614 17731 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.558904 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.564975 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:40.568128 16481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:40.568828 16350 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:40.573974 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.586720 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.672803 16370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9e2c42ca67f4fea9e8dc724658c2090"
dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48538
I20240501 20:08:40.673027 16370 raft_consensus.cc:595] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:40.769331 17731 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.783654 16481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:40.789909 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.805652 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.806243 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:40.824077 16350 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:40.858865 17861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.905966 19012 raft_consensus.cc:979] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: : Instructing follower cc976bfc5fd643c8bb631f097245c65f to start an election
I20240501 20:08:40.906164 19012 raft_consensus.cc:1067] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Signalling peer cc976bfc5fd643c8bb631f097245c65f to start an election
I20240501 20:08:40.907130 17881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e2c42ca67f4fea9e8dc724658c2090"
dest_uuid: "cc976bfc5fd643c8bb631f097245c65f"
from {username='slave'} at 127.13.251.129:42183
I20240501 20:08:40.907465 17881 raft_consensus.cc:482] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:40.907725 17881 raft_consensus.cc:3037] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:40.910881 17881 raft_consensus.cc:504] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:40.912531 17881 leader_election.cc:288] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [CANDIDATE]: Term 2 election: Requested vote from peers b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:40.913093 16370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e2c42ca67f4fea9e8dc724658c2090" candidate_uuid: "cc976bfc5fd643c8bb631f097245c65f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
I20240501 20:08:40.913327 16370 raft_consensus.cc:3032] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:40.913401 16370 raft_consensus.cc:729] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:40.913647 16370 consensus_queue.cc:260] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:40.913893 16370 raft_consensus.cc:3037] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:40.915836 16370 raft_consensus.cc:2454] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc976bfc5fd643c8bb631f097245c65f in term 2.
I20240501 20:08:40.916445 17814 leader_election.cc:302] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc, cc976bfc5fd643c8bb631f097245c65f; no voters:
I20240501 20:08:40.917193 19013 raft_consensus.cc:2781] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:40.917845 19013 raft_consensus.cc:686] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 2 LEADER]: Becoming Leader. State: Replica: cc976bfc5fd643c8bb631f097245c65f, State: Running, Role: LEADER
I20240501 20:08:40.918462 19013 consensus_queue.cc:237] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:40.927037 16221 catalog_manager.cc:5484] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f reported cstate change: term changed from 1 to 2, leader changed from b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) to cc976bfc5fd643c8bb631f097245c65f (127.13.251.133). New cstate: current_term: 2 leader_uuid: "cc976bfc5fd643c8bb631f097245c65f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:41.033841 16501 raft_consensus.cc:1261] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Refusing update from remote peer 6427cd62d4294811a5b4fd1cbd81d448: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:41.034535 18942 consensus_queue.cc:1035] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:41.049631 17731 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.050082 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.061691 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.069238 17861 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.069741 16350 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:41.071789 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:41.112088 16481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:41.164278 16370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "609385e741e04820bb903f2664108ff2"
dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48570
I20240501 20:08:41.164527 16370 raft_consensus.cc:595] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:41.259600 17731 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.274749 16481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:41.282397 17861 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.284711 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.285416 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.287075 16350 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:41.292883 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:41.488922 17731 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.495645 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.499017 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.504715 16481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:41.519050 16370 raft_consensus.cc:1261] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Refusing update from remote peer cc976bfc5fd643c8bb631f097245c65f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:41.520406 19129 consensus_queue.cc:1035] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:41.522460 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:41.534045 16350 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:41.561414 17861 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.563900 19012 raft_consensus.cc:979] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: : Instructing follower 10d2f39dd2434b8e9b9d683defc79425 to start an election
I20240501 20:08:41.564175 19012 raft_consensus.cc:1067] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Signalling peer 10d2f39dd2434b8e9b9d683defc79425 to start an election
I20240501 20:08:41.565023 17751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "609385e741e04820bb903f2664108ff2"
dest_uuid: "10d2f39dd2434b8e9b9d683defc79425"
from {username='slave'} at 127.13.251.129:48575
I20240501 20:08:41.565337 17751 raft_consensus.cc:482] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:41.565488 17751 raft_consensus.cc:3037] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:41.569677 17751 raft_consensus.cc:504] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:41.571350 17751 leader_election.cc:288] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [CANDIDATE]: Term 2 election: Requested vote from peers b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:41.571499 16369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609385e741e04820bb903f2664108ff2" candidate_uuid: "10d2f39dd2434b8e9b9d683defc79425" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
I20240501 20:08:41.571807 16369 raft_consensus.cc:3032] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:41.571934 16369 raft_consensus.cc:729] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:41.572269 16369 consensus_queue.cc:260] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:41.572850 16369 raft_consensus.cc:3037] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:41.577188 16369 raft_consensus.cc:2454] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10d2f39dd2434b8e9b9d683defc79425 in term 2.
I20240501 20:08:41.578075 17684 leader_election.cc:302] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 10d2f39dd2434b8e9b9d683defc79425, b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:41.578696 18926 raft_consensus.cc:2781] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:41.599886 18926 raft_consensus.cc:686] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 2 LEADER]: Becoming Leader. State: Replica: 10d2f39dd2434b8e9b9d683defc79425, State: Running, Role: LEADER
I20240501 20:08:41.600353 18926 consensus_queue.cc:237] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:41.606242 16221 catalog_manager.cc:5484] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 reported cstate change: term changed from 1 to 2, leader changed from b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) to 10d2f39dd2434b8e9b9d683defc79425 (127.13.251.132). New cstate: current_term: 2 leader_uuid: "10d2f39dd2434b8e9b9d683defc79425" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:41.654630 16501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb81d991e8a3455db9f94b6ef07b86ee"
dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45960
I20240501 20:08:41.654896 16501 raft_consensus.cc:595] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:41.754105 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.756953 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:41.759501 16481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:41.769749 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:41.780346 16350 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:41.780757 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:41.782829 17861 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.007426 18133 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.009392 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.011734 16481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:08:42.018015 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.018712 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.037528 17861 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.041074 16350 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:42.078181 16370 raft_consensus.cc:1261] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Refusing update from remote peer 10d2f39dd2434b8e9b9d683defc79425: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:42.078931 18926 consensus_queue.cc:1035] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:42.100056 19217 raft_consensus.cc:979] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: : Instructing follower 3c7f7a1988b749d6a2bfa4d374e87e69 to start an election
I20240501 20:08:42.100350 18802 raft_consensus.cc:1067] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Signalling peer 3c7f7a1988b749d6a2bfa4d374e87e69 to start an election
I20240501 20:08:42.101424 18153 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb81d991e8a3455db9f94b6ef07b86ee"
dest_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69"
from {username='slave'} at 127.13.251.130:45763
I20240501 20:08:42.101668 18153 raft_consensus.cc:482] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:42.101773 18153 raft_consensus.cc:3037] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:42.105124 18153 raft_consensus.cc:504] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:42.109560 18153 leader_election.cc:288] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [CANDIDATE]: Term 2 election: Requested vote from peers 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:42.110102 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb81d991e8a3455db9f94b6ef07b86ee" candidate_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
I20240501 20:08:42.110354 16501 raft_consensus.cc:3032] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:42.110466 16501 raft_consensus.cc:729] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:42.110718 16501 consensus_queue.cc:260] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:42.110949 16501 raft_consensus.cc:3037] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:42.113957 16501 raft_consensus.cc:2454] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c7f7a1988b749d6a2bfa4d374e87e69 in term 2.
I20240501 20:08:42.114627 18088 leader_election.cc:302] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3c7f7a1988b749d6a2bfa4d374e87e69, 63a2e807217a4182a7f7e5a1a105b376; no voters:
I20240501 20:08:42.115083 19153 raft_consensus.cc:2781] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:42.117394 19153 raft_consensus.cc:686] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 2 LEADER]: Becoming Leader. State: Replica: 3c7f7a1988b749d6a2bfa4d374e87e69, State: Running, Role: LEADER
I20240501 20:08:42.117760 19153 consensus_queue.cc:237] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:42.121734 16224 catalog_manager.cc:5484] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 reported cstate change: term changed from 1 to 2, leader changed from 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) to 3c7f7a1988b749d6a2bfa4d374e87e69 (127.13.251.135). New cstate: current_term: 2 leader_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:42.132493 16370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "699e4265f7d14e47b55ab17594554f77"
dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48620
I20240501 20:08:42.132751 16370 raft_consensus.cc:595] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:42.235422 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.239936 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.244236 17992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.250715 16481 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:42.254684 16350 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:42.271616 17861 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.271775 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.358937 19266 raft_consensus.cc:979] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: : Instructing follower 474b2d91e93f46be855e2bf1243bf0c8 to start an election
I20240501 20:08:42.359119 19266 raft_consensus.cc:1067] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Signalling peer 474b2d91e93f46be855e2bf1243bf0c8 to start an election
I20240501 20:08:42.360167 18013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "699e4265f7d14e47b55ab17594554f77"
dest_uuid: "474b2d91e93f46be855e2bf1243bf0c8"
from {username='slave'} at 127.13.251.129:36051
I20240501 20:08:42.360484 18013 raft_consensus.cc:482] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:42.360628 18013 raft_consensus.cc:3037] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:42.364904 18013 raft_consensus.cc:504] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:42.367534 18013 leader_election.cc:288] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [CANDIDATE]: Term 2 election: Requested vote from peers b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:42.367590 16370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699e4265f7d14e47b55ab17594554f77" candidate_uuid: "474b2d91e93f46be855e2bf1243bf0c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
I20240501 20:08:42.367848 16370 raft_consensus.cc:3032] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:42.367944 16370 raft_consensus.cc:729] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:42.368239 16370 consensus_queue.cc:260] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:42.368494 16370 raft_consensus.cc:3037] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:42.371147 16370 raft_consensus.cc:2454] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 474b2d91e93f46be855e2bf1243bf0c8 in term 2.
I20240501 20:08:42.371793 17944 leader_election.cc:302] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474b2d91e93f46be855e2bf1243bf0c8, b75e08cafe834ce580d6666f3a22bdfc; no voters:
I20240501 20:08:42.372772 19269 raft_consensus.cc:2781] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:42.373917 19269 raft_consensus.cc:686] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 2 LEADER]: Becoming Leader. State: Replica: 474b2d91e93f46be855e2bf1243bf0c8, State: Running, Role: LEADER
I20240501 20:08:42.374406 19269 consensus_queue.cc:237] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:42.382949 16221 catalog_manager.cc:5484] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 reported cstate change: term changed from 1 to 2, leader changed from b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) to 474b2d91e93f46be855e2bf1243bf0c8 (127.13.251.134). New cstate: current_term: 2 leader_uuid: "474b2d91e93f46be855e2bf1243bf0c8" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } } }
I20240501 20:08:42.489651 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.493367 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.494611 16481 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:42.494894 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.532979 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.556360 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:42.565503 17861 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.628481 16501 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b380bf3c85748cfb4843ae21eadc74b"
dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46016
I20240501 20:08:42.628683 16501 raft_consensus.cc:595] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:42.675521 16501 raft_consensus.cc:1261] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Refusing update from remote peer 3c7f7a1988b749d6a2bfa4d374e87e69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:42.676224 19153 consensus_queue.cc:1035] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:42.729135 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.731848 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.738306 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.758585 16481 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:42.767875 17861 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.772173 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:42.775017 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:42.890971 16370 raft_consensus.cc:1261] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Refusing update from remote peer 474b2d91e93f46be855e2bf1243bf0c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:42.891696 19343 consensus_queue.cc:1035] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:43.163704 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.185441 16481 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:08:43.190357 17861 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:43.190919 19385 raft_consensus.cc:979] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: : Instructing follower cc976bfc5fd643c8bb631f097245c65f to start an election
I20240501 20:08:43.191197 19217 raft_consensus.cc:1067] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Signalling peer cc976bfc5fd643c8bb631f097245c65f to start an election
I20240501 20:08:43.191385 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.191905 17880 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0b380bf3c85748cfb4843ae21eadc74b"
dest_uuid: "cc976bfc5fd643c8bb631f097245c65f"
from {username='slave'} at 127.13.251.130:59843
I20240501 20:08:43.192138 17880 raft_consensus.cc:482] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:43.192289 17880 raft_consensus.cc:3037] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:43.197469 17880 raft_consensus.cc:504] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:43.198829 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.199667 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b380bf3c85748cfb4843ae21eadc74b" candidate_uuid: "cc976bfc5fd643c8bb631f097245c65f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
I20240501 20:08:43.200074 17880 leader_election.cc:288] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [CANDIDATE]: Term 2 election: Requested vote from peers 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:43.200887 16501 raft_consensus.cc:3032] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:43.201005 16501 raft_consensus.cc:729] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:43.201354 16501 consensus_queue.cc:260] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:43.201676 16501 raft_consensus.cc:3037] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:43.206409 16501 raft_consensus.cc:2454] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc976bfc5fd643c8bb631f097245c65f in term 2.
I20240501 20:08:43.207376 17816 leader_election.cc:302] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376, cc976bfc5fd643c8bb631f097245c65f; no voters:
I20240501 20:08:43.207861 19349 raft_consensus.cc:2781] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:43.211022 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.222381 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.237386 19349 raft_consensus.cc:686] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 2 LEADER]: Becoming Leader. State: Replica: cc976bfc5fd643c8bb631f097245c65f, State: Running, Role: LEADER
I20240501 20:08:43.237850 19349 consensus_queue.cc:237] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:43.254300 16228 catalog_manager.cc:5484] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f reported cstate change: term changed from 1 to 2, leader changed from 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) to cc976bfc5fd643c8bb631f097245c65f (127.13.251.133). New cstate: current_term: 2 leader_uuid: "cc976bfc5fd643c8bb631f097245c65f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:43.333437 18153 consensus_queue.cc:237] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:43.334751 19153 raft_consensus.cc:2932] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:43.340240 16228 catalog_manager.cc:5484] T f522ede49dac4f7193158c6bca98ed26 P 3c7f7a1988b749d6a2bfa4d374e87e69 reported cstate change: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New cstate: current_term: 2 leader_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:43.344748 16350 tablet_service.cc:1514] Processing DeleteTablet for tablet f522ede49dac4f7193158c6bca98ed26 with delete_type TABLET_DATA_TOMBSTONED (TS b75e08cafe834ce580d6666f3a22bdfc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60160
I20240501 20:08:43.345574 19415 tablet_replica.cc:330] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: stopping tablet replica
I20240501 20:08:43.345975 19415 raft_consensus.cc:2229] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:43.346313 19415 raft_consensus.cc:2258] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:43.347476 19415 ts_tablet_manager.cc:1875] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:43.353425 19415 ts_tablet_manager.cc:1888] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:43.353571 19415 log.cc:1196] T f522ede49dac4f7193158c6bca98ed26 P b75e08cafe834ce580d6666f3a22bdfc: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/f522ede49dac4f7193158c6bca98ed26
I20240501 20:08:43.354502 16213 catalog_manager.cc:4830] TS b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281): tablet f522ede49dac4f7193158c6bca98ed26 (table rebalance_test_table_0 [id=850cf659e4ff4f46aa0f6b6664d00a27]) successfully deleted
I20240501 20:08:43.439692 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.472568 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.472599 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.477497 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.480620 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.484743 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.520165 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.607718 17751 consensus_queue.cc:237] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:43.609215 19218 raft_consensus.cc:2932] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:43.614444 16228 catalog_manager.cc:5484] T 608ac70c00014df693e8bd48925af0c3 P 10d2f39dd2434b8e9b9d683defc79425 reported cstate change: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New cstate: current_term: 2 leader_uuid: "10d2f39dd2434b8e9b9d683defc79425" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:43.618324 16481 tablet_service.cc:1514] Processing DeleteTablet for tablet 608ac70c00014df693e8bd48925af0c3 with delete_type TABLET_DATA_TOMBSTONED (TS 63a2e807217a4182a7f7e5a1a105b376 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47658
I20240501 20:08:43.619254 19470 tablet_replica.cc:330] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: stopping tablet replica
I20240501 20:08:43.619748 19470 raft_consensus.cc:2229] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:43.620074 19470 raft_consensus.cc:2258] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:43.621536 19470 ts_tablet_manager.cc:1875] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:43.628310 19470 ts_tablet_manager.cc:1888] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:43.628489 19470 log.cc:1196] T 608ac70c00014df693e8bd48925af0c3 P 63a2e807217a4182a7f7e5a1a105b376: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/608ac70c00014df693e8bd48925af0c3
I20240501 20:08:43.629455 16215 catalog_manager.cc:4830] TS 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035): tablet 608ac70c00014df693e8bd48925af0c3 (table rebalance_test_table_3 [id=4e128faf8de54db1923732a8c5a70f5a]) successfully deleted
I20240501 20:08:43.698590 16501 raft_consensus.cc:1261] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Refusing update from remote peer cc976bfc5fd643c8bb631f097245c65f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:43.699775 19349 consensus_queue.cc:1035] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:43.722465 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.736186 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.737881 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.738358 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.740084 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.743042 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:43.765204 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:43.872505 17620 consensus_queue.cc:237] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:43.873759 19054 raft_consensus.cc:2932] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:43.879029 16224 catalog_manager.cc:5484] T 3acfb78e3127445db565fc5ceaca8b9e P 6427cd62d4294811a5b4fd1cbd81d448 reported cstate change: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New cstate: current_term: 2 leader_uuid: "6427cd62d4294811a5b4fd1cbd81d448" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:43.882661 16350 tablet_service.cc:1514] Processing DeleteTablet for tablet 3acfb78e3127445db565fc5ceaca8b9e with delete_type TABLET_DATA_TOMBSTONED (TS b75e08cafe834ce580d6666f3a22bdfc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60160
I20240501 20:08:43.883280 19528 tablet_replica.cc:330] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: stopping tablet replica
I20240501 20:08:43.883517 19528 raft_consensus.cc:2229] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:43.883783 19528 raft_consensus.cc:2258] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:43.884622 19528 ts_tablet_manager.cc:1875] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:43.890388 19528 ts_tablet_manager.cc:1888] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:43.890501 19528 log.cc:1196] T 3acfb78e3127445db565fc5ceaca8b9e P b75e08cafe834ce580d6666f3a22bdfc: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/3acfb78e3127445db565fc5ceaca8b9e
I20240501 20:08:43.891170 16213 catalog_manager.cc:4830] TS b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281): tablet 3acfb78e3127445db565fc5ceaca8b9e (table rebalance_test_table_2 [id=357f26ae29a643e3b064aeb9bcfc24ab]) successfully deleted
I20240501 20:08:43.991597 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.011387 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:44.020514 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.025050 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.026199 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.029848 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:44.032801 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.138515 17620 consensus_queue.cc:237] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } }
I20240501 20:08:44.139755 19054 raft_consensus.cc:2932] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:44.144403 16228 catalog_manager.cc:5484] T eda61e86d34d4ee687006c3cae7a51bf P 6427cd62d4294811a5b4fd1cbd81d448 reported cstate change: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New cstate: current_term: 2 leader_uuid: "6427cd62d4294811a5b4fd1cbd81d448" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6427cd62d4294811a5b4fd1cbd81d448" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 45387 } } }
I20240501 20:08:44.148300 16481 tablet_service.cc:1514] Processing DeleteTablet for tablet eda61e86d34d4ee687006c3cae7a51bf with delete_type TABLET_DATA_TOMBSTONED (TS 63a2e807217a4182a7f7e5a1a105b376 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47658
I20240501 20:08:44.149021 19581 tablet_replica.cc:330] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: stopping tablet replica
I20240501 20:08:44.149297 19581 raft_consensus.cc:2229] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:44.149547 19581 raft_consensus.cc:2258] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:44.150523 19581 ts_tablet_manager.cc:1875] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:44.157462 19581 ts_tablet_manager.cc:1888] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:44.157683 19581 log.cc:1196] T eda61e86d34d4ee687006c3cae7a51bf P 63a2e807217a4182a7f7e5a1a105b376: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/eda61e86d34d4ee687006c3cae7a51bf
I20240501 20:08:44.158416 16215 catalog_manager.cc:4830] TS 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035): tablet eda61e86d34d4ee687006c3cae7a51bf (table rebalance_test_table_4 [id=4083fed5a42341cb8f80a011df1f5d16]) successfully deleted
I20240501 20:08:44.259244 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.272758 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.287403 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:44.292594 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.297461 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.330090 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:44.349673 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.470510 17880 consensus_queue.cc:237] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:44.471894 19349 raft_consensus.cc:2932] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:44.476394 16223 catalog_manager.cc:5484] T c9e2c42ca67f4fea9e8dc724658c2090 P cc976bfc5fd643c8bb631f097245c65f reported cstate change: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New cstate: current_term: 2 leader_uuid: "cc976bfc5fd643c8bb631f097245c65f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:44.480701 16350 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e2c42ca67f4fea9e8dc724658c2090 with delete_type TABLET_DATA_TOMBSTONED (TS b75e08cafe834ce580d6666f3a22bdfc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60160
I20240501 20:08:44.481514 19635 tablet_replica.cc:330] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: stopping tablet replica
I20240501 20:08:44.481725 19635 raft_consensus.cc:2229] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:44.481940 19635 raft_consensus.cc:2258] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:44.482878 19635 ts_tablet_manager.cc:1875] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:44.489794 19635 ts_tablet_manager.cc:1888] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:44.489946 19635 log.cc:1196] T c9e2c42ca67f4fea9e8dc724658c2090 P b75e08cafe834ce580d6666f3a22bdfc: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/c9e2c42ca67f4fea9e8dc724658c2090
I20240501 20:08:44.490854 16213 catalog_manager.cc:4830] TS b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281): tablet c9e2c42ca67f4fea9e8dc724658c2090 (table rebalance_test_table_4 [id=4083fed5a42341cb8f80a011df1f5d16]) successfully deleted
I20240501 20:08:44.630960 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.633261 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.639248 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.652653 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.682585 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:44.698994 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:44.745747 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.823283 17751 consensus_queue.cc:237] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } }
I20240501 20:08:44.824790 19688 raft_consensus.cc:2932] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:44.829641 16224 catalog_manager.cc:5484] T 609385e741e04820bb903f2664108ff2 P 10d2f39dd2434b8e9b9d683defc79425 reported cstate change: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New cstate: current_term: 2 leader_uuid: "10d2f39dd2434b8e9b9d683defc79425" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10d2f39dd2434b8e9b9d683defc79425" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 43563 } } }
I20240501 20:08:44.833441 16350 tablet_service.cc:1514] Processing DeleteTablet for tablet 609385e741e04820bb903f2664108ff2 with delete_type TABLET_DATA_TOMBSTONED (TS b75e08cafe834ce580d6666f3a22bdfc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60160
I20240501 20:08:44.833863 19635 tablet_replica.cc:330] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: stopping tablet replica
I20240501 20:08:44.834053 19635 raft_consensus.cc:2229] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:44.834270 19635 raft_consensus.cc:2258] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:44.834941 19635 ts_tablet_manager.cc:1875] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:44.839429 19635 ts_tablet_manager.cc:1888] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:44.839718 19635 log.cc:1196] T 609385e741e04820bb903f2664108ff2 P b75e08cafe834ce580d6666f3a22bdfc: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/609385e741e04820bb903f2664108ff2
I20240501 20:08:44.840612 16213 catalog_manager.cc:4830] TS b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281): tablet 609385e741e04820bb903f2664108ff2 (table rebalance_test_table_2 [id=357f26ae29a643e3b064aeb9bcfc24ab]) successfully deleted
I20240501 20:08:44.949236 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.958663 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.967980 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:44.981155 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.000272 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.018396 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.021520 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.110891 18153 consensus_queue.cc:237] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } }
I20240501 20:08:45.112246 19601 raft_consensus.cc:2932] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:45.116679 16224 catalog_manager.cc:5484] T fb81d991e8a3455db9f94b6ef07b86ee P 3c7f7a1988b749d6a2bfa4d374e87e69 reported cstate change: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New cstate: current_term: 2 leader_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3c7f7a1988b749d6a2bfa4d374e87e69" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 34297 } } }
I20240501 20:08:45.120652 16481 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81d991e8a3455db9f94b6ef07b86ee with delete_type TABLET_DATA_TOMBSTONED (TS 63a2e807217a4182a7f7e5a1a105b376 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47658
I20240501 20:08:45.121850 19748 tablet_replica.cc:330] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: stopping tablet replica
I20240501 20:08:45.122031 19748 raft_consensus.cc:2229] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:45.122233 19748 raft_consensus.cc:2258] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:45.122807 19748 ts_tablet_manager.cc:1875] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:45.128629 19748 ts_tablet_manager.cc:1888] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:45.128747 19748 log.cc:1196] T fb81d991e8a3455db9f94b6ef07b86ee P 63a2e807217a4182a7f7e5a1a105b376: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/fb81d991e8a3455db9f94b6ef07b86ee
I20240501 20:08:45.129619 16215 catalog_manager.cc:4830] TS 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035): tablet fb81d991e8a3455db9f94b6ef07b86ee (table rebalance_test_table_1 [id=373549282c7d4d6c98c887b22195d023]) successfully deleted
I20240501 20:08:45.222075 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.222649 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.226522 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.239025 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.247507 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.257436 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.274369 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.359320 18013 consensus_queue.cc:237] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } }
I20240501 20:08:45.360755 19578 raft_consensus.cc:2932] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } } }
I20240501 20:08:45.365624 16228 catalog_manager.cc:5484] T 699e4265f7d14e47b55ab17594554f77 P 474b2d91e93f46be855e2bf1243bf0c8 reported cstate change: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New cstate: current_term: 2 leader_uuid: "474b2d91e93f46be855e2bf1243bf0c8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "474b2d91e93f46be855e2bf1243bf0c8" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 40765 } } }
I20240501 20:08:45.369383 16350 tablet_service.cc:1514] Processing DeleteTablet for tablet 699e4265f7d14e47b55ab17594554f77 with delete_type TABLET_DATA_TOMBSTONED (TS b75e08cafe834ce580d6666f3a22bdfc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60160
I20240501 20:08:45.370138 19801 tablet_replica.cc:330] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: stopping tablet replica
I20240501 20:08:45.370476 19801 raft_consensus.cc:2229] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:45.370726 19801 raft_consensus.cc:2258] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:45.371799 19801 ts_tablet_manager.cc:1875] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:45.378943 19801 ts_tablet_manager.cc:1888] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:45.379050 19801 log.cc:1196] T 699e4265f7d14e47b55ab17594554f77 P b75e08cafe834ce580d6666f3a22bdfc: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/699e4265f7d14e47b55ab17594554f77
I20240501 20:08:45.379765 16213 catalog_manager.cc:4830] TS b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281): tablet 699e4265f7d14e47b55ab17594554f77 (table rebalance_test_table_0 [id=850cf659e4ff4f46aa0f6b6664d00a27]) successfully deleted
I20240501 20:08:45.495040 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.496757 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.513149 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.521740 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.531270 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.532994 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:45.557847 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:45.635706 17880 consensus_queue.cc:237] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } }
I20240501 20:08:45.636957 19779 raft_consensus.cc:2932] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:45.641845 16229 catalog_manager.cc:5484] T 0b380bf3c85748cfb4843ae21eadc74b P cc976bfc5fd643c8bb631f097245c65f reported cstate change: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New cstate: current_term: 2 leader_uuid: "cc976bfc5fd643c8bb631f097245c65f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc976bfc5fd643c8bb631f097245c65f" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 41735 } } }
I20240501 20:08:45.645285 16481 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b380bf3c85748cfb4843ae21eadc74b with delete_type TABLET_DATA_TOMBSTONED (TS 63a2e807217a4182a7f7e5a1a105b376 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47658
I20240501 20:08:45.645859 19854 tablet_replica.cc:330] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: stopping tablet replica
I20240501 20:08:45.646045 19854 raft_consensus.cc:2229] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:45.646235 19854 raft_consensus.cc:2258] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:45.646821 19854 ts_tablet_manager.cc:1875] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:45.651273 19854 ts_tablet_manager.cc:1888] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:45.651386 19854 log.cc:1196] T 0b380bf3c85748cfb4843ae21eadc74b P 63a2e807217a4182a7f7e5a1a105b376: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/0b380bf3c85748cfb4843ae21eadc74b
I20240501 20:08:45.652179 16215 catalog_manager.cc:4830] TS 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035): tablet 0b380bf3c85748cfb4843ae21eadc74b (table rebalance_test_table_3 [id=4e128faf8de54db1923732a8c5a70f5a]) successfully deleted
I20240501 20:08:46.024303 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.052024 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:46.053721 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.056581 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.072978 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.073453 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:46.073853 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.250012 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.260103 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:46.273509 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.277251 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.278872 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.280534 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:46.299176 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:47.126749 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.131419 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.162546 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.166168 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:47.169095 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.211031 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.218755 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:47.400125 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.403643 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.414984 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:47.415146 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:47.417114 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.440332 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.457468 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.731492 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.751598 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:47.754503 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.755571 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.764549 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.772186 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:47.792119 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
919cd28c68d8439dbca079e3b938e719 | 127.13.251.190:43847 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.251.148:34359 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
10d2f39dd2434b8e9b9d683defc79425 | 127.13.251.132:43563 | HEALTHY | <none> | 2 | 0
3c7f7a1988b749d6a2bfa4d374e87e69 | 127.13.251.135:34297 | HEALTHY | <none> | 2 | 0
474b2d91e93f46be855e2bf1243bf0c8 | 127.13.251.134:40765 | HEALTHY | <none> | 2 | 0
63a2e807217a4182a7f7e5a1a105b376 | 127.13.251.130:41035 | HEALTHY | <none> | 3 | 0
6427cd62d4294811a5b4fd1cbd81d448 | 127.13.251.131:45387 | HEALTHY | <none> | 2 | 0
b75e08cafe834ce580d6666f3a22bdfc | 127.13.251.129:44281 | HEALTHY | <none> | 3 | 0
cc976bfc5fd643c8bb631f097245c65f | 127.13.251.133:41735 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.13.251.129 | experimental | 127.13.251.129:44281
local_ip_for_outbound_sockets | 127.13.251.130 | experimental | 127.13.251.130:41035
local_ip_for_outbound_sockets | 127.13.251.131 | experimental | 127.13.251.131:45387
local_ip_for_outbound_sockets | 127.13.251.132 | experimental | 127.13.251.132:43563
local_ip_for_outbound_sockets | 127.13.251.133 | experimental | 127.13.251.133:41735
local_ip_for_outbound_sockets | 127.13.251.134 | experimental | 127.13.251.134:40765
local_ip_for_outbound_sockets | 127.13.251.135 | experimental | 127.13.251.135:34297
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.13.251.129:44281
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.13.251.130:41035
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.13.251.131:45387
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.13.251.132:43563
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.13.251.133:41735
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.13.251.134:40765
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.13.251.135:34297
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.251.148:34359 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:08:47.860061 14318 log_verifier.cc:126] Checking tablet 08f747bc44c94917920cd5b84a4ac356
I20240501 20:08:47.875620 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 08f747bc44c94917920cd5b84a4ac356
I20240501 20:08:47.875789 14318 log_verifier.cc:126] Checking tablet 09d70c138c8240b394fa6bdb0d3018d9
I20240501 20:08:47.883569 14318 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 09d70c138c8240b394fa6bdb0d3018d9
I20240501 20:08:47.883702 14318 log_verifier.cc:126] Checking tablet 0b380bf3c85748cfb4843ae21eadc74b
I20240501 20:08:47.896129 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0b380bf3c85748cfb4843ae21eadc74b
I20240501 20:08:47.896236 14318 log_verifier.cc:126] Checking tablet 317b376708eb4b268b93fb8d1c15258a
I20240501 20:08:47.910971 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 317b376708eb4b268b93fb8d1c15258a
I20240501 20:08:47.911100 14318 log_verifier.cc:126] Checking tablet 33eb5bad3a7e48559804a50b09085802
I20240501 20:08:47.928225 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 33eb5bad3a7e48559804a50b09085802
I20240501 20:08:47.928347 14318 log_verifier.cc:126] Checking tablet 3acfb78e3127445db565fc5ceaca8b9e
I20240501 20:08:47.941380 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3acfb78e3127445db565fc5ceaca8b9e
I20240501 20:08:47.941491 14318 log_verifier.cc:126] Checking tablet 44f3b394dd644f1da3821d53dd7c735b
I20240501 20:08:47.953677 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 44f3b394dd644f1da3821d53dd7c735b
I20240501 20:08:47.953783 14318 log_verifier.cc:126] Checking tablet 608ac70c00014df693e8bd48925af0c3
I20240501 20:08:47.966871 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 608ac70c00014df693e8bd48925af0c3
I20240501 20:08:47.966982 14318 log_verifier.cc:126] Checking tablet 609385e741e04820bb903f2664108ff2
I20240501 20:08:47.979382 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 609385e741e04820bb903f2664108ff2
I20240501 20:08:47.979493 14318 log_verifier.cc:126] Checking tablet 699e4265f7d14e47b55ab17594554f77
I20240501 20:08:47.992081 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 699e4265f7d14e47b55ab17594554f77
I20240501 20:08:47.992189 14318 log_verifier.cc:126] Checking tablet 70db4cb78b6f4256b4752601bc730808
I20240501 20:08:48.004071 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 70db4cb78b6f4256b4752601bc730808
I20240501 20:08:48.004173 14318 log_verifier.cc:126] Checking tablet 99cfaa6da58247f0b477bf2cbe1aea51
I20240501 20:08:48.016247 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 99cfaa6da58247f0b477bf2cbe1aea51
I20240501 20:08:48.016356 14318 log_verifier.cc:126] Checking tablet c9e2c42ca67f4fea9e8dc724658c2090
I20240501 20:08:48.028792 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c9e2c42ca67f4fea9e8dc724658c2090
I20240501 20:08:48.028911 14318 log_verifier.cc:126] Checking tablet eda61e86d34d4ee687006c3cae7a51bf
I20240501 20:08:48.041025 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet eda61e86d34d4ee687006c3cae7a51bf
I20240501 20:08:48.041162 14318 log_verifier.cc:126] Checking tablet f522ede49dac4f7193158c6bca98ed26
I20240501 20:08:48.053270 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f522ede49dac4f7193158c6bca98ed26
I20240501 20:08:48.053412 14318 log_verifier.cc:126] Checking tablet fb81d991e8a3455db9f94b6ef07b86ee
I20240501 20:08:48.066133 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet fb81d991e8a3455db9f94b6ef07b86ee
I20240501 20:08:48.069317 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.136:0
--local_ip_for_outbound_sockets=127.13.251.136
--webserver_interface=127.13.251.136
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--builtin_ntp_servers=127.13.251.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:08:48.422729 20083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:48.423053 20083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:48.423193 20083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:48.432739 20083 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:48.432911 20083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:48.433095 20083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.136
I20240501 20:08:48.448679 20083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.13.251.136
--webserver_port=0
--tserver_master_addrs=127.13.251.190:43847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.136
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:48.450068 20083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:48.451902 20083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:48.465759 20088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:48.466401 20089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:48.467237 20091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:48.467926 20083 server_base.cc:1034] running on GCE node
I20240501 20:08:48.468585 20083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:48.469589 20083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:48.470811 20083 hybrid_clock.cc:648] HybridClock initialized: now 1714594128470740 us; error 59 us; skew 500 ppm
I20240501 20:08:48.471217 20083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:48.473719 20083 webserver.cc:460] Webserver started at http://127.13.251.136:40505/ using document root <none> and password file <none>
I20240501 20:08:48.474457 20083 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:48.474558 20083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:48.474843 20083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:48.477722 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c6d6d8fb9d1c49e9b3e661830d798b42"
format_stamp: "Formatted at 2024-05-01 20:08:48 on dist-test-slave-qswb"
I20240501 20:08:48.478515 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c6d6d8fb9d1c49e9b3e661830d798b42"
format_stamp: "Formatted at 2024-05-01 20:08:48 on dist-test-slave-qswb"
I20240501 20:08:48.484424 20083 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240501 20:08:48.487916 20097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:48.489442 20083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 20:08:48.489589 20083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c6d6d8fb9d1c49e9b3e661830d798b42"
format_stamp: "Formatted at 2024-05-01 20:08:48 on dist-test-slave-qswb"
I20240501 20:08:48.489769 20083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:48.514307 20083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:48.515187 20083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:48.515429 20083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:48.516364 20083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:48.518330 20083 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:48.518416 20083 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:48.518496 20083 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:48.518548 20083 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:48.556444 20083 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.136:44765
I20240501 20:08:48.556514 20209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.136:44765 every 8 connection(s)
I20240501 20:08:48.557801 20083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:08:48.560266 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 20083
I20240501 20:08:48.560514 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:08:48.571893 20210 heartbeater.cc:347] Connected to a master server at 127.13.251.190:43847
I20240501 20:08:48.572203 20210 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:48.573000 20210 heartbeater.cc:511] Master 127.13.251.190:43847 requested a full tablet report, sending...
I20240501 20:08:48.574549 16221 ts_manager.cc:194] Registered new tserver with Master: c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136:44765)
I20240501 20:08:48.575811 16221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.136:34555
I20240501 20:08:49.072263 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.087636 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.095261 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.107885 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:49.109495 20145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.118674 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:49.119412 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.135833 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.340343 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.340857 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.351030 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.379878 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.392818 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:49.404929 20145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.414654 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:49.420019 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:49.532845 16370 consensus_queue.cc:237] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
W20240501 20:08:49.541328 16303 consensus_peers.cc:485] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc -> Peer c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136:44765): Couldn't send request to peer c6d6d8fb9d1c49e9b3e661830d798b42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08f747bc44c94917920cd5b84a4ac356. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:49.578449 20210 heartbeater.cc:503] Master 127.13.251.190:43847 was elected leader, sending a full tablet report...
I20240501 20:08:49.954290 20315 ts_tablet_manager.cc:902] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: Initiating tablet copy from peer b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:49.956218 20315 tablet_copy_client.cc:308] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:44281
I20240501 20:08:49.963090 16390 tablet_copy_service.cc:140] P b75e08cafe834ce580d6666f3a22bdfc: Received BeginTabletCopySession request for tablet 08f747bc44c94917920cd5b84a4ac356 from peer c6d6d8fb9d1c49e9b3e661830d798b42 ({username='slave'} at 127.13.251.136:52331)
I20240501 20:08:49.963290 16390 tablet_copy_service.cc:161] P b75e08cafe834ce580d6666f3a22bdfc: Beginning new tablet copy session on tablet 08f747bc44c94917920cd5b84a4ac356 from peer c6d6d8fb9d1c49e9b3e661830d798b42 at {username='slave'} at 127.13.251.136:52331: session id = c6d6d8fb9d1c49e9b3e661830d798b42-08f747bc44c94917920cd5b84a4ac356
I20240501 20:08:49.965497 16390 tablet_copy_source_session.cc:205] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:49.969967 20315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f747bc44c94917920cd5b84a4ac356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:49.979650 20315 tablet_copy_client.cc:783] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:49.979964 20315 tablet_copy_client.cc:647] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:49.982021 20315 tablet_copy_client.cc:523] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:49.985090 20315 tablet_bootstrap.cc:492] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: Bootstrap starting.
I20240501 20:08:49.995083 20315 log.cc:824] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:50.027767 20315 tablet_bootstrap.cc:492] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:50.028277 20315 tablet_bootstrap.cc:492] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: Bootstrap complete.
I20240501 20:08:50.028671 20315 ts_tablet_manager.cc:1367] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: Time spent bootstrapping tablet: real 0.044s user 0.031s sys 0.013s
I20240501 20:08:50.033375 20315 raft_consensus.cc:348] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } }
I20240501 20:08:50.034158 20315 raft_consensus.cc:729] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d6d8fb9d1c49e9b3e661830d798b42, State: Initialized, Role: FOLLOWER
I20240501 20:08:50.034601 20315 consensus_queue.cc:260] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:50.035995 20315 ts_tablet_manager.cc:1398] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20240501 20:08:50.037040 16390 tablet_copy_service.cc:342] P b75e08cafe834ce580d6666f3a22bdfc: Request end of tablet copy session c6d6d8fb9d1c49e9b3e661830d798b42-08f747bc44c94917920cd5b84a4ac356 received from {username='slave'} at 127.13.251.136:52331
I20240501 20:08:50.037149 16390 tablet_copy_service.cc:433] P b75e08cafe834ce580d6666f3a22bdfc: ending tablet copy session c6d6d8fb9d1c49e9b3e661830d798b42-08f747bc44c94917920cd5b84a4ac356 on tablet 08f747bc44c94917920cd5b84a4ac356 with peer c6d6d8fb9d1c49e9b3e661830d798b42
I20240501 20:08:50.379299 20165 raft_consensus.cc:1203] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:50.381006 20323 raft_consensus.cc:2932] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:50.381909 20165 raft_consensus.cc:2932] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:50.386216 16224 catalog_manager.cc:5484] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc reported cstate change: config changed from index -1 to 4, VOTER c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "b75e08cafe834ce580d6666f3a22bdfc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:50.416821 16500 consensus_queue.cc:237] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
W20240501 20:08:50.425504 16434 consensus_peers.cc:485] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 -> Peer c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136:44765): Couldn't send request to peer c6d6d8fb9d1c49e9b3e661830d798b42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 317b376708eb4b268b93fb8d1c15258a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:50.832674 20350 ts_tablet_manager.cc:902] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: Initiating tablet copy from peer 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:50.833783 20350 tablet_copy_client.cc:308] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:41035
I20240501 20:08:50.839910 16521 tablet_copy_service.cc:140] P 63a2e807217a4182a7f7e5a1a105b376: Received BeginTabletCopySession request for tablet 317b376708eb4b268b93fb8d1c15258a from peer c6d6d8fb9d1c49e9b3e661830d798b42 ({username='slave'} at 127.13.251.136:48059)
I20240501 20:08:50.840068 16521 tablet_copy_service.cc:161] P 63a2e807217a4182a7f7e5a1a105b376: Beginning new tablet copy session on tablet 317b376708eb4b268b93fb8d1c15258a from peer c6d6d8fb9d1c49e9b3e661830d798b42 at {username='slave'} at 127.13.251.136:48059: session id = c6d6d8fb9d1c49e9b3e661830d798b42-317b376708eb4b268b93fb8d1c15258a
I20240501 20:08:50.842096 16521 tablet_copy_source_session.cc:205] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:50.843394 20350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 317b376708eb4b268b93fb8d1c15258a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:50.847712 20350 tablet_copy_client.cc:783] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:50.848057 20350 tablet_copy_client.cc:647] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:50.850373 20350 tablet_copy_client.cc:523] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:50.852859 20350 tablet_bootstrap.cc:492] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: Bootstrap starting.
I20240501 20:08:50.882515 20350 tablet_bootstrap.cc:492] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:08:50.882948 20350 tablet_bootstrap.cc:492] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: Bootstrap complete.
I20240501 20:08:50.883194 20350 ts_tablet_manager.cc:1367] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20240501 20:08:50.884014 20350 raft_consensus.cc:348] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } }
I20240501 20:08:50.884430 20350 raft_consensus.cc:729] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d6d8fb9d1c49e9b3e661830d798b42, State: Initialized, Role: FOLLOWER
I20240501 20:08:50.884577 20350 consensus_queue.cc:260] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:50.884939 20350 ts_tablet_manager.cc:1398] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:08:50.885710 16521 tablet_copy_service.cc:342] P 63a2e807217a4182a7f7e5a1a105b376: Request end of tablet copy session c6d6d8fb9d1c49e9b3e661830d798b42-317b376708eb4b268b93fb8d1c15258a received from {username='slave'} at 127.13.251.136:48059
I20240501 20:08:50.885835 16521 tablet_copy_service.cc:433] P 63a2e807217a4182a7f7e5a1a105b376: ending tablet copy session c6d6d8fb9d1c49e9b3e661830d798b42-317b376708eb4b268b93fb8d1c15258a on tablet 317b376708eb4b268b93fb8d1c15258a with peer c6d6d8fb9d1c49e9b3e661830d798b42
I20240501 20:08:51.404438 20165 raft_consensus.cc:1203] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:08:51.405869 20360 raft_consensus.cc:2932] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:51.406522 20165 raft_consensus.cc:2932] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:51.412207 16224 catalog_manager.cc:5484] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 reported cstate change: config changed from index -1 to 4, VOTER c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "63a2e807217a4182a7f7e5a1a105b376" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:51.517719 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.534571 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:51.536227 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.539650 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.571627 20145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:51.574960 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.586724 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.601824 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:51.662324 16500 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "317b376708eb4b268b93fb8d1c15258a"
dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51416
I20240501 20:08:51.662526 16500 raft_consensus.cc:595] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:51.757069 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.771709 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.777503 16481 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:51.781411 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.807364 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.817549 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:51.824630 20145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:51.826918 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:51.922770 20457 raft_consensus.cc:979] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: : Instructing follower c6d6d8fb9d1c49e9b3e661830d798b42 to start an election
I20240501 20:08:51.922938 20457 raft_consensus.cc:1067] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Signalling peer c6d6d8fb9d1c49e9b3e661830d798b42 to start an election
I20240501 20:08:51.924015 20165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "317b376708eb4b268b93fb8d1c15258a"
dest_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42"
from {username='slave'} at 127.13.251.130:51483
I20240501 20:08:51.924314 20165 raft_consensus.cc:482] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:51.924521 20165 raft_consensus.cc:3037] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:51.926843 20165 raft_consensus.cc:504] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:51.928494 20165 leader_election.cc:288] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [CANDIDATE]: Term 2 election: Requested vote from peers 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035)
I20240501 20:08:51.936028 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "317b376708eb4b268b93fb8d1c15258a" candidate_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63a2e807217a4182a7f7e5a1a105b376"
I20240501 20:08:51.936271 16500 raft_consensus.cc:3032] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:51.936349 16500 raft_consensus.cc:729] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63a2e807217a4182a7f7e5a1a105b376, State: Running, Role: LEADER
I20240501 20:08:51.936676 16500 consensus_queue.cc:260] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:51.936950 16500 raft_consensus.cc:3037] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:51.938969 16500 raft_consensus.cc:2454] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d6d8fb9d1c49e9b3e661830d798b42 in term 2.
I20240501 20:08:51.939709 20100 leader_election.cc:302] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 63a2e807217a4182a7f7e5a1a105b376, c6d6d8fb9d1c49e9b3e661830d798b42; no voters:
I20240501 20:08:51.948539 20465 raft_consensus.cc:2781] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:51.949081 20465 raft_consensus.cc:686] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 2 LEADER]: Becoming Leader. State: Replica: c6d6d8fb9d1c49e9b3e661830d798b42, State: Running, Role: LEADER
I20240501 20:08:51.949672 20465 consensus_queue.cc:237] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:51.953595 16224 catalog_manager.cc:5484] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 reported cstate change: term changed from 1 to 2, leader changed from 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) to c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136). New cstate: current_term: 2 leader_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:52.028486 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.044057 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.045244 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.063958 20145 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:52.070243 16481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.070365 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.074133 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:52.111960 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.173983 16370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08f747bc44c94917920cd5b84a4ac356"
dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39848
I20240501 20:08:52.174239 16370 raft_consensus.cc:595] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Received request to transfer leadership
I20240501 20:08:52.275077 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.286984 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.287151 16481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.290010 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.311416 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.313108 16350 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:52.315881 20145 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:52.345438 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.347863 20535 raft_consensus.cc:979] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: : Instructing follower c6d6d8fb9d1c49e9b3e661830d798b42 to start an election
I20240501 20:08:52.348233 20458 raft_consensus.cc:1067] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Signalling peer c6d6d8fb9d1c49e9b3e661830d798b42 to start an election
I20240501 20:08:52.349207 20165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08f747bc44c94917920cd5b84a4ac356"
dest_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42"
from {username='slave'} at 127.13.251.129:54053
I20240501 20:08:52.349444 20165 raft_consensus.cc:482] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:52.349610 20165 raft_consensus.cc:3037] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:52.352818 20165 raft_consensus.cc:504] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:52.384725 16370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f747bc44c94917920cd5b84a4ac356" candidate_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b75e08cafe834ce580d6666f3a22bdfc"
I20240501 20:08:52.385016 16370 raft_consensus.cc:3032] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:52.385110 16370 raft_consensus.cc:729] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b75e08cafe834ce580d6666f3a22bdfc, State: Running, Role: LEADER
I20240501 20:08:52.385401 16370 consensus_queue.cc:260] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:52.385630 16370 raft_consensus.cc:3037] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:52.388099 20165 leader_election.cc:288] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [CANDIDATE]: Term 2 election: Requested vote from peers b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281)
I20240501 20:08:52.388538 16370 raft_consensus.cc:2454] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d6d8fb9d1c49e9b3e661830d798b42 in term 2.
I20240501 20:08:52.389071 20098 leader_election.cc:302] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b75e08cafe834ce580d6666f3a22bdfc, c6d6d8fb9d1c49e9b3e661830d798b42; no voters:
I20240501 20:08:52.389436 20465 raft_consensus.cc:2781] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:52.389631 20465 raft_consensus.cc:686] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 2 LEADER]: Becoming Leader. State: Replica: c6d6d8fb9d1c49e9b3e661830d798b42, State: Running, Role: LEADER
I20240501 20:08:52.389911 20465 consensus_queue.cc:237] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:52.393427 16220 catalog_manager.cc:5484] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 reported cstate change: term changed from 1 to 2, leader changed from b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) to c6d6d8fb9d1c49e9b3e661830d798b42 (127.13.251.136). New cstate: current_term: 2 leader_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 } } peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:52.454072 16500 raft_consensus.cc:1261] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Refusing update from remote peer c6d6d8fb9d1c49e9b3e661830d798b42: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:52.454911 20465 consensus_queue.cc:1035] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a2e807217a4182a7f7e5a1a105b376" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 41035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:52.751072 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.767331 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.775414 16481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.775951 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.778493 16350 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.792519 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.817456 20145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.853708 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:52.888763 16370 raft_consensus.cc:1261] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Refusing update from remote peer c6d6d8fb9d1c49e9b3e661830d798b42: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:08:52.889639 20466 consensus_queue.cc:1035] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75e08cafe834ce580d6666f3a22bdfc" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:52.951992 20165 consensus_queue.cc:237] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:52.952914 20465 raft_consensus.cc:2932] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:52.957250 16223 catalog_manager.cc:5484] T 317b376708eb4b268b93fb8d1c15258a P c6d6d8fb9d1c49e9b3e661830d798b42 reported cstate change: config changed from index 4 to 6, VOTER 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130) evicted. New cstate: current_term: 2 leader_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:52.962723 16481 tablet_service.cc:1514] Processing DeleteTablet for tablet 317b376708eb4b268b93fb8d1c15258a with delete_type TABLET_DATA_TOMBSTONED (TS 63a2e807217a4182a7f7e5a1a105b376 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47658
I20240501 20:08:52.963707 20609 tablet_replica.cc:330] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: stopping tablet replica
I20240501 20:08:52.963981 20609 raft_consensus.cc:2229] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:52.964241 20609 raft_consensus.cc:2258] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:52.966318 20609 ts_tablet_manager.cc:1875] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:52.971012 20609 ts_tablet_manager.cc:1888] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:52.971145 20609 log.cc:1196] T 317b376708eb4b268b93fb8d1c15258a P 63a2e807217a4182a7f7e5a1a105b376: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/317b376708eb4b268b93fb8d1c15258a
I20240501 20:08:52.972172 16215 catalog_manager.cc:4830] TS 63a2e807217a4182a7f7e5a1a105b376 (127.13.251.130:41035): tablet 317b376708eb4b268b93fb8d1c15258a (table rebalance_test_table_1 [id=373549282c7d4d6c98c887b22195d023]) successfully deleted
I20240501 20:08:53.061501 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.071585 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.083595 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.085569 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.113291 16481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.121786 20145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.128674 16350 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.131842 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.231912 20165 consensus_queue.cc:237] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } }
I20240501 20:08:53.232779 20465 raft_consensus.cc:2932] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:53.237097 16223 catalog_manager.cc:5484] T 08f747bc44c94917920cd5b84a4ac356 P c6d6d8fb9d1c49e9b3e661830d798b42 reported cstate change: config changed from index 4 to 6, VOTER b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129) evicted. New cstate: current_term: 2 leader_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c6d6d8fb9d1c49e9b3e661830d798b42" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44765 } } }
I20240501 20:08:53.240310 16350 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f747bc44c94917920cd5b84a4ac356 with delete_type TABLET_DATA_TOMBSTONED (TS b75e08cafe834ce580d6666f3a22bdfc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60160
I20240501 20:08:53.240967 20662 tablet_replica.cc:330] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: stopping tablet replica
I20240501 20:08:53.241147 20662 raft_consensus.cc:2229] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:53.241331 20662 raft_consensus.cc:2258] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:53.242859 20662 ts_tablet_manager.cc:1875] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:53.247154 20662 ts_tablet_manager.cc:1888] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:08:53.247265 20662 log.cc:1196] T 08f747bc44c94917920cd5b84a4ac356 P b75e08cafe834ce580d6666f3a22bdfc: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/08f747bc44c94917920cd5b84a4ac356
I20240501 20:08:53.248030 16213 catalog_manager.cc:4830] TS b75e08cafe834ce580d6666f3a22bdfc (127.13.251.129:44281): tablet 08f747bc44c94917920cd5b84a4ac356 (table rebalance_test_table_4 [id=4083fed5a42341cb8f80a011df1f5d16]) successfully deleted
I20240501 20:08:53.554189 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.582710 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.582849 16481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.583972 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.586503 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.589159 20145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.605302 16350 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.612272 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.786717 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.794186 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.810506 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.819415 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.820701 16481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.826779 16350 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.827138 20145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:53.884527 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:54.185948 17731 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:54.198235 17992 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:54.199411 18133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:54.213618 16481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:54.234355 17600 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:54.235870 17861 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:54.235877 16350 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:08:54.236958 20145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
919cd28c68d8439dbca079e3b938e719 | 127.13.251.190:43847 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.251.148:34359 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
10d2f39dd2434b8e9b9d683defc79425 | 127.13.251.132:43563 | HEALTHY | <none> | 2 | 0
3c7f7a1988b749d6a2bfa4d374e87e69 | 127.13.251.135:34297 | HEALTHY | <none> | 2 | 0
474b2d91e93f46be855e2bf1243bf0c8 | 127.13.251.134:40765 | HEALTHY | <none> | 2 | 0
63a2e807217a4182a7f7e5a1a105b376 | 127.13.251.130:41035 | HEALTHY | <none> | 2 | 0
6427cd62d4294811a5b4fd1cbd81d448 | 127.13.251.131:45387 | HEALTHY | <none> | 2 | 0
b75e08cafe834ce580d6666f3a22bdfc | 127.13.251.129:44281 | HEALTHY | <none> | 2 | 0
c6d6d8fb9d1c49e9b3e661830d798b42 | 127.13.251.136:44765 | HEALTHY | <none> | 2 | 0
cc976bfc5fd643c8bb631f097245c65f | 127.13.251.133:41735 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.13.251.129 | experimental | 127.13.251.129:44281
local_ip_for_outbound_sockets | 127.13.251.130 | experimental | 127.13.251.130:41035
local_ip_for_outbound_sockets | 127.13.251.131 | experimental | 127.13.251.131:45387
local_ip_for_outbound_sockets | 127.13.251.132 | experimental | 127.13.251.132:43563
local_ip_for_outbound_sockets | 127.13.251.133 | experimental | 127.13.251.133:41735
local_ip_for_outbound_sockets | 127.13.251.134 | experimental | 127.13.251.134:40765
local_ip_for_outbound_sockets | 127.13.251.135 | experimental | 127.13.251.135:34297
local_ip_for_outbound_sockets | 127.13.251.136 | experimental | 127.13.251.136:44765
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.13.251.129:44281
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.13.251.130:41035
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.13.251.131:45387
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.13.251.132:43563
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.13.251.133:41735
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.13.251.134:40765
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.13.251.135:34297
server_dump_info_path | /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.13.251.136:44765
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.251.148:34359 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:08:54.323302 14318 log_verifier.cc:126] Checking tablet 08f747bc44c94917920cd5b84a4ac356
I20240501 20:08:54.336983 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 08f747bc44c94917920cd5b84a4ac356
I20240501 20:08:54.337103 14318 log_verifier.cc:126] Checking tablet 09d70c138c8240b394fa6bdb0d3018d9
I20240501 20:08:54.342648 14318 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 09d70c138c8240b394fa6bdb0d3018d9
I20240501 20:08:54.342747 14318 log_verifier.cc:126] Checking tablet 0b380bf3c85748cfb4843ae21eadc74b
I20240501 20:08:54.352952 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0b380bf3c85748cfb4843ae21eadc74b
I20240501 20:08:54.353050 14318 log_verifier.cc:126] Checking tablet 317b376708eb4b268b93fb8d1c15258a
I20240501 20:08:54.365901 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 317b376708eb4b268b93fb8d1c15258a
I20240501 20:08:54.366021 14318 log_verifier.cc:126] Checking tablet 33eb5bad3a7e48559804a50b09085802
I20240501 20:08:54.375201 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 33eb5bad3a7e48559804a50b09085802
I20240501 20:08:54.375291 14318 log_verifier.cc:126] Checking tablet 3acfb78e3127445db565fc5ceaca8b9e
I20240501 20:08:54.385425 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3acfb78e3127445db565fc5ceaca8b9e
I20240501 20:08:54.385540 14318 log_verifier.cc:126] Checking tablet 44f3b394dd644f1da3821d53dd7c735b
I20240501 20:08:54.395380 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 44f3b394dd644f1da3821d53dd7c735b
I20240501 20:08:54.395489 14318 log_verifier.cc:126] Checking tablet 608ac70c00014df693e8bd48925af0c3
I20240501 20:08:54.405979 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 608ac70c00014df693e8bd48925af0c3
I20240501 20:08:54.406082 14318 log_verifier.cc:126] Checking tablet 609385e741e04820bb903f2664108ff2
I20240501 20:08:54.416476 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 609385e741e04820bb903f2664108ff2
I20240501 20:08:54.416642 14318 log_verifier.cc:126] Checking tablet 699e4265f7d14e47b55ab17594554f77
I20240501 20:08:54.427194 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 699e4265f7d14e47b55ab17594554f77
I20240501 20:08:54.427301 14318 log_verifier.cc:126] Checking tablet 70db4cb78b6f4256b4752601bc730808
I20240501 20:08:54.440529 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 70db4cb78b6f4256b4752601bc730808
I20240501 20:08:54.440647 14318 log_verifier.cc:126] Checking tablet 99cfaa6da58247f0b477bf2cbe1aea51
I20240501 20:08:54.454844 14318 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 99cfaa6da58247f0b477bf2cbe1aea51
I20240501 20:08:54.454937 14318 log_verifier.cc:126] Checking tablet c9e2c42ca67f4fea9e8dc724658c2090
I20240501 20:08:54.465375 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c9e2c42ca67f4fea9e8dc724658c2090
I20240501 20:08:54.465476 14318 log_verifier.cc:126] Checking tablet eda61e86d34d4ee687006c3cae7a51bf
I20240501 20:08:54.475685 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet eda61e86d34d4ee687006c3cae7a51bf
I20240501 20:08:54.475781 14318 log_verifier.cc:126] Checking tablet f522ede49dac4f7193158c6bca98ed26
I20240501 20:08:54.485536 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f522ede49dac4f7193158c6bca98ed26
I20240501 20:08:54.485641 14318 log_verifier.cc:126] Checking tablet fb81d991e8a3455db9f94b6ef07b86ee
I20240501 20:08:54.496173 14318 log_verifier.cc:177] Verified matching terms for 6 ops in tablet fb81d991e8a3455db9f94b6ef07b86ee
I20240501 20:08:54.497139 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 16288
I20240501 20:08:54.498754 16409 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:54.693938 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 16288
I20240501 20:08:54.728013 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 16419
I20240501 20:08:54.728982 16541 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:54.924222 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 16419
I20240501 20:08:54.959107 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 17538
I20240501 20:08:54.965555 17659 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.138840 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 17538
I20240501 20:08:55.169245 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 17669
I20240501 20:08:55.175325 17791 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.348229 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 17669
I20240501 20:08:55.377935 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 17799
I20240501 20:08:55.383956 17920 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.556610 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 17799
I20240501 20:08:55.586345 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 17929
I20240501 20:08:55.592198 18056 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.762332 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 17929
I20240501 20:08:55.791679 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 18067
I20240501 20:08:55.797889 18193 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.972362 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 18067
I20240501 20:08:56.001997 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 20083
I20240501 20:08:56.007692 20204 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:56.117318 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 20083
I20240501 20:08:56.137850 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 16198
I20240501 20:08:56.138927 16259 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:56.364404 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 16198
2024-05-01T20:08:56Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (45110 ms)
[----------] 1 test from RebalanceParamTest (45110 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240501 20:08:56.448055 14318 test_util.cc:268] Using random seed: 242211869
I20240501 20:08:56.449999 14318 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:56.450058 14318 ts_itest-base.cc:113] --------------
I20240501 20:08:56.450099 14318 ts_itest-base.cc:114] 8 tablet servers
I20240501 20:08:56.450150 14318 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:08:56.450203 14318 ts_itest-base.cc:116] --------------
2024-05-01T20:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:56Z Disabled control of system clock
I20240501 20:08:56.481007 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.251.190:44663
--webserver_interface=127.13.251.190
--webserver_port=0
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.251.190:44663
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 20:08:56.817116 20795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:56.817443 20795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:56.817546 20795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:56.825220 20795 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:56.825359 20795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:56.825444 20795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:56.825516 20795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:56.825578 20795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:56.834825 20795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.13.251.190:44663
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.251.190:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.251.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:56.836241 20795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:56.838002 20795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:56.849850 20801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:56.850605 20800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:56.851253 20803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:56.851946 20795 server_base.cc:1034] running on GCE node
I20240501 20:08:56.852777 20795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:56.854380 20795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:56.855631 20795 hybrid_clock.cc:648] HybridClock initialized: now 1714594136855581 us; error 35 us; skew 500 ppm
I20240501 20:08:56.855996 20795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:56.858270 20795 webserver.cc:460] Webserver started at http://127.13.251.190:36169/ using document root <none> and password file <none>
I20240501 20:08:56.859032 20795 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:56.859146 20795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:56.859452 20795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:56.862185 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-qswb"
I20240501 20:08:56.862973 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-qswb"
I20240501 20:08:56.868292 20795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:08:56.871770 20809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.873310 20795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:56.873471 20795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-qswb"
I20240501 20:08:56.873670 20795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:56.902736 20795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:56.903645 20795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:56.903910 20795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:56.922298 20795 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.190:44663
I20240501 20:08:56.922329 20860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.190:44663 every 8 connection(s)
I20240501 20:08:56.923681 20795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:56.927377 20861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:56.932732 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 20795
I20240501 20:08:56.932962 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:56.938015 20861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Bootstrap starting.
I20240501 20:08:56.941109 20861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:56.942153 20861 log.cc:824] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:56.944908 20861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: No bootstrap required, opened a new log
I20240501 20:08:56.951354 20861 raft_consensus.cc:348] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 44663 } }
I20240501 20:08:56.951722 20861 raft_consensus.cc:374] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:56.951830 20861 raft_consensus.cc:729] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f878915f05c04c2c8c7cd9cf39cd6cf8, State: Initialized, Role: FOLLOWER
I20240501 20:08:56.952391 20861 consensus_queue.cc:260] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 44663 } }
I20240501 20:08:56.952572 20861 raft_consensus.cc:388] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:56.952663 20861 raft_consensus.cc:482] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:56.952777 20861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:56.954743 20861 raft_consensus.cc:504] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 44663 } }
I20240501 20:08:56.955225 20861 leader_election.cc:302] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f878915f05c04c2c8c7cd9cf39cd6cf8; no voters:
I20240501 20:08:56.955811 20861 leader_election.cc:288] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:56.955937 20866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:56.957165 20866 raft_consensus.cc:686] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [term 1 LEADER]: Becoming Leader. State: Replica: f878915f05c04c2c8c7cd9cf39cd6cf8, State: Running, Role: LEADER
I20240501 20:08:56.957748 20866 consensus_queue.cc:237] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 44663 } }
I20240501 20:08:56.957939 20861 sys_catalog.cc:564] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:56.960804 20868 sys_catalog.cc:455] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f878915f05c04c2c8c7cd9cf39cd6cf8. Latest consensus state: current_term: 1 leader_uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 44663 } } }
I20240501 20:08:56.962972 20868 sys_catalog.cc:458] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:56.960810 20867 sys_catalog.cc:455] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f878915f05c04c2c8c7cd9cf39cd6cf8" member_type: VOTER last_known_addr { host: "127.13.251.190" port: 44663 } } }
I20240501 20:08:56.963634 20867 sys_catalog.cc:458] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:56.966192 20873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:56.970966 20873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:56.979220 20873 catalog_manager.cc:1347] Generated new cluster ID: 48502957b56445c98da4234ebb8dbe9a
I20240501 20:08:56.979332 20873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:56.992710 20873 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:56.994323 20873 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:57.018064 20873 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Generated new TSK 0
I20240501 20:08:57.019202 20873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:57.030293 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.129:0
--local_ip_for_outbound_sockets=127.13.251.129
--webserver_interface=127.13.251.129
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:08:57.371439 20885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:57.371778 20885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:57.371894 20885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:57.379671 20885 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:57.379793 20885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:57.379916 20885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.129
I20240501 20:08:57.388965 20885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.251.129
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.129
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:57.390218 20885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:57.391906 20885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:57.404222 20890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:57.404775 20893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:57.405685 20891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:57.406594 20885 server_base.cc:1034] running on GCE node
I20240501 20:08:57.407212 20885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:57.408195 20885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:57.409425 20885 hybrid_clock.cc:648] HybridClock initialized: now 1714594137409378 us; error 36 us; skew 500 ppm
I20240501 20:08:57.409746 20885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:57.412019 20885 webserver.cc:460] Webserver started at http://127.13.251.129:39049/ using document root <none> and password file <none>
I20240501 20:08:57.412820 20885 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:57.412967 20885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:57.413276 20885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:57.417917 20885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cda19d6a81c0473db302535ce2ada7f0"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-qswb"
I20240501 20:08:57.418689 20885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cda19d6a81c0473db302535ce2ada7f0"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-qswb"
I20240501 20:08:57.423991 20885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:08:57.427520 20899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.429003 20885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:57.429183 20885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cda19d6a81c0473db302535ce2ada7f0"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-qswb"
I20240501 20:08:57.429402 20885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:57.468462 20885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:57.469341 20885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:57.469626 20885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:57.470609 20885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:57.472193 20885 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:57.472291 20885 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.472419 20885 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:57.472491 20885 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.503657 20885 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.129:41259
I20240501 20:08:57.503736 21011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.129:41259 every 8 connection(s)
I20240501 20:08:57.504976 20885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:57.511823 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 20885
I20240501 20:08:57.512029 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:57.515241 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.130:0
--local_ip_for_outbound_sockets=127.13.251.130
--webserver_interface=127.13.251.130
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:57.518229 21012 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:08:57.518573 21012 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:57.519517 21012 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:08:57.521875 20826 ts_manager.cc:194] Registered new tserver with Master: cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:08:57.523828 20826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.129:56239
W20240501 20:08:57.854081 21016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:57.854398 21016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:57.854552 21016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:57.862537 21016 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:57.862680 21016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:57.862861 21016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.130
I20240501 20:08:57.872449 21016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.251.130
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.130
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:57.873881 21016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:57.875674 21016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:57.888497 21022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:57.889204 21024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:57.889919 21021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:57.890641 21016 server_base.cc:1034] running on GCE node
I20240501 20:08:57.891240 21016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:57.892256 21016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:57.893443 21016 hybrid_clock.cc:648] HybridClock initialized: now 1714594137893387 us; error 54 us; skew 500 ppm
I20240501 20:08:57.893766 21016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:57.895995 21016 webserver.cc:460] Webserver started at http://127.13.251.130:38131/ using document root <none> and password file <none>
I20240501 20:08:57.896706 21016 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:57.896806 21016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:57.897105 21016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:57.899708 21016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "feac73db0a3b4aafa374ca5dfd322b08"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-qswb"
I20240501 20:08:57.900424 21016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "feac73db0a3b4aafa374ca5dfd322b08"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-qswb"
I20240501 20:08:57.905345 21016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:57.908782 21030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.910066 21016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:57.910221 21016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "feac73db0a3b4aafa374ca5dfd322b08"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-qswb"
I20240501 20:08:57.910410 21016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:57.932595 21016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:57.933445 21016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:57.933723 21016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:57.934796 21016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:57.936385 21016 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:57.936491 21016 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.936604 21016 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:57.936672 21016 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.969692 21016 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.130:42271
I20240501 20:08:57.969744 21142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.130:42271 every 8 connection(s)
I20240501 20:08:57.971024 21016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:57.978680 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 21016
I20240501 20:08:57.978863 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:57.982175 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.131:0
--local_ip_for_outbound_sockets=127.13.251.131
--webserver_interface=127.13.251.131
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:57.984305 21143 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:08:57.984612 21143 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:57.985350 21143 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:08:57.986775 20826 ts_manager.cc:194] Registered new tserver with Master: feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:08:57.987893 20826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.130:53237
W20240501 20:08:58.322329 21146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:58.322659 21146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:58.322855 21146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:58.330789 21146 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:58.330971 21146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:58.331154 21146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.131
I20240501 20:08:58.341018 21146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.251.131
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.131
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:58.342499 21146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:58.344436 21146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:58.357421 21151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:58.358140 21146 server_base.cc:1034] running on GCE node
W20240501 20:08:58.358551 21154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:58.359349 21152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:58.360041 21146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:58.361017 21146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:58.362262 21146 hybrid_clock.cc:648] HybridClock initialized: now 1714594138362201 us; error 41 us; skew 500 ppm
I20240501 20:08:58.362624 21146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:58.365115 21146 webserver.cc:460] Webserver started at http://127.13.251.131:35611/ using document root <none> and password file <none>
I20240501 20:08:58.365902 21146 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:58.366006 21146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:58.366326 21146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:58.369170 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4af096f5a6a9440a94f6682b399efe20"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-qswb"
I20240501 20:08:58.370048 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4af096f5a6a9440a94f6682b399efe20"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-qswb"
I20240501 20:08:58.375921 21146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 20:08:58.379693 21160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.381160 21146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:58.381397 21146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4af096f5a6a9440a94f6682b399efe20"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-qswb"
I20240501 20:08:58.381625 21146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:58.403293 21146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:58.404212 21146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:58.404475 21146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:58.405517 21146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:58.407034 21146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:58.407140 21146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.407250 21146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:58.407327 21146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.439911 21146 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.131:38349
I20240501 20:08:58.439956 21272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.131:38349 every 8 connection(s)
I20240501 20:08:58.441282 21146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:58.445756 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 21146
I20240501 20:08:58.445945 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:58.448968 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.132:0
--local_ip_for_outbound_sockets=127.13.251.132
--webserver_interface=127.13.251.132
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:58.454605 21273 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:08:58.455049 21273 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:58.456121 21273 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:08:58.457522 20826 ts_manager.cc:194] Registered new tserver with Master: 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:08:58.458411 20826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.131:42955
I20240501 20:08:58.526808 21012 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
W20240501 20:08:58.779517 21276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:58.779865 21276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:58.780002 21276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:58.787633 21276 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:58.787778 21276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:58.787935 21276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.132
I20240501 20:08:58.797016 21276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.13.251.132
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.132
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:58.798385 21276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:58.800194 21276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:58.812446 21282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:58.813311 21284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:58.813858 21281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:58.814566 21276 server_base.cc:1034] running on GCE node
I20240501 20:08:58.815197 21276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:58.816136 21276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:58.817348 21276 hybrid_clock.cc:648] HybridClock initialized: now 1714594138817308 us; error 31 us; skew 500 ppm
I20240501 20:08:58.817653 21276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:58.819895 21276 webserver.cc:460] Webserver started at http://127.13.251.132:39199/ using document root <none> and password file <none>
I20240501 20:08:58.820609 21276 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:58.820717 21276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:58.821053 21276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:58.823670 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "61380023001a4ee9ada9ec2c6c1b7159"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-qswb"
I20240501 20:08:58.824425 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "61380023001a4ee9ada9ec2c6c1b7159"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-qswb"
I20240501 20:08:58.829525 21276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:58.832901 21290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.834301 21276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:58.834458 21276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "61380023001a4ee9ada9ec2c6c1b7159"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-qswb"
I20240501 20:08:58.834646 21276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:58.857606 21276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:58.858541 21276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:58.858811 21276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:58.859908 21276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:58.861625 21276 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:58.861723 21276 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.861850 21276 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:58.861925 21276 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.891836 21276 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.132:46399
I20240501 20:08:58.891901 21402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.132:46399 every 8 connection(s)
I20240501 20:08:58.893139 21276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:58.901301 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 21276
I20240501 20:08:58.901448 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:58.904595 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.133:0
--local_ip_for_outbound_sockets=127.13.251.133
--webserver_interface=127.13.251.133
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:58.906122 21403 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:08:58.906455 21403 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:58.907368 21403 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:08:58.908983 20826 ts_manager.cc:194] Registered new tserver with Master: 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:08:58.909870 20826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.132:57247
I20240501 20:08:58.990918 21143 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
W20240501 20:08:59.238260 21406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:59.238597 21406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:59.238741 21406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:59.246256 21406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:59.246412 21406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:59.246582 21406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.133
I20240501 20:08:59.256047 21406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.13.251.133
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.133
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:59.257402 21406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:59.259219 21406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:59.271873 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:59.272668 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:59.273337 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:59.274050 21406 server_base.cc:1034] running on GCE node
I20240501 20:08:59.274665 21406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:59.275652 21406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:59.276854 21406 hybrid_clock.cc:648] HybridClock initialized: now 1714594139276810 us; error 39 us; skew 500 ppm
I20240501 20:08:59.277164 21406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:59.279410 21406 webserver.cc:460] Webserver started at http://127.13.251.133:46189/ using document root <none> and password file <none>
I20240501 20:08:59.280153 21406 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:59.280275 21406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:59.280575 21406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:59.283145 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
format_stamp: "Formatted at 2024-05-01 20:08:59 on dist-test-slave-qswb"
I20240501 20:08:59.283937 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
format_stamp: "Formatted at 2024-05-01 20:08:59 on dist-test-slave-qswb"
I20240501 20:08:59.289161 21406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:59.292454 21420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:59.293737 21406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:59.293893 21406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
format_stamp: "Formatted at 2024-05-01 20:08:59 on dist-test-slave-qswb"
I20240501 20:08:59.294088 21406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:59.316237 21406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:59.317103 21406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:59.317379 21406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:59.318322 21406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:59.319830 21406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:59.319927 21406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:59.320050 21406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:59.320135 21406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:59.352121 21406 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.133:37155
I20240501 20:08:59.352180 21532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.133:37155 every 8 connection(s)
I20240501 20:08:59.353394 21406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:59.357489 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 21406
I20240501 20:08:59.357715 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:59.361116 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.134:0
--local_ip_for_outbound_sockets=127.13.251.134
--webserver_interface=127.13.251.134
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:59.366626 21533 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:08:59.367012 21533 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:59.367890 21533 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:08:59.369272 20826 ts_manager.cc:194] Registered new tserver with Master: 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:08:59.370046 20826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.133:32979
I20240501 20:08:59.460986 21273 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
W20240501 20:08:59.688702 21536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:59.688994 21536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:59.689132 21536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:59.696723 21536 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:08:59.696862 21536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:59.697036 21536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.134
I20240501 20:08:59.707002 21536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.13.251.134
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.134
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:59.708407 21536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:59.710238 21536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:59.723248 21541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:59.723850 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:59.724752 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:59.725602 21536 server_base.cc:1034] running on GCE node
I20240501 20:08:59.726186 21536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:59.727110 21536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:59.728323 21536 hybrid_clock.cc:648] HybridClock initialized: now 1714594139728276 us; error 48 us; skew 500 ppm
I20240501 20:08:59.728636 21536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:59.731009 21536 webserver.cc:460] Webserver started at http://127.13.251.134:34981/ using document root <none> and password file <none>
I20240501 20:08:59.731809 21536 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:59.731940 21536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:59.732251 21536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:59.734915 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
format_stamp: "Formatted at 2024-05-01 20:08:59 on dist-test-slave-qswb"
I20240501 20:08:59.735788 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
format_stamp: "Formatted at 2024-05-01 20:08:59 on dist-test-slave-qswb"
I20240501 20:08:59.741209 21536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:59.744710 21550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:59.746124 21536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:59.746289 21536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
format_stamp: "Formatted at 2024-05-01 20:08:59 on dist-test-slave-qswb"
I20240501 20:08:59.746488 21536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:59.767925 21536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:59.768801 21536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:59.769079 21536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:59.770033 21536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:59.771562 21536 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:59.771678 21536 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:59.771783 21536 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:59.771840 21536 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:59.805243 21536 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.134:42431
I20240501 20:08:59.805289 21662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.134:42431 every 8 connection(s)
I20240501 20:08:59.806548 21536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:08:59.812804 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 21536
I20240501 20:08:59.813028 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:08:59.816635 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.135:0
--local_ip_for_outbound_sockets=127.13.251.135
--webserver_interface=127.13.251.135
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:08:59.820773 21663 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:08:59.821162 21663 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:59.822096 21663 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:08:59.823619 20826 ts_manager.cc:194] Registered new tserver with Master: c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:08:59.824422 20826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.134:44063
I20240501 20:08:59.912533 21403 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
W20240501 20:09:00.154013 21666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:00.154301 21666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:00.154387 21666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:00.161877 21666 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:00.162003 21666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:00.162122 21666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.135
I20240501 20:09:00.171131 21666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.13.251.135
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.135
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:00.172446 21666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:00.174070 21666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:00.186031 21672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:00.186547 21671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:00.187374 21674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:00.188174 21666 server_base.cc:1034] running on GCE node
I20240501 20:09:00.188709 21666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:00.189637 21666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:00.190837 21666 hybrid_clock.cc:648] HybridClock initialized: now 1714594140190789 us; error 53 us; skew 500 ppm
I20240501 20:09:00.191145 21666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:00.193571 21666 webserver.cc:460] Webserver started at http://127.13.251.135:39293/ using document root <none> and password file <none>
I20240501 20:09:00.194414 21666 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:00.194535 21666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:00.194854 21666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:00.197445 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "744c5a532e724391830c0785d32eb2fb"
format_stamp: "Formatted at 2024-05-01 20:09:00 on dist-test-slave-qswb"
I20240501 20:09:00.198246 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "744c5a532e724391830c0785d32eb2fb"
format_stamp: "Formatted at 2024-05-01 20:09:00 on dist-test-slave-qswb"
I20240501 20:09:00.203323 21666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 20:09:00.206622 21680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:00.207967 21666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:00.208117 21666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "744c5a532e724391830c0785d32eb2fb"
format_stamp: "Formatted at 2024-05-01 20:09:00 on dist-test-slave-qswb"
I20240501 20:09:00.208312 21666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:00.245707 21666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:00.246616 21666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:00.246901 21666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:00.247900 21666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:00.249430 21666 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:00.249534 21666 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:00.249648 21666 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:00.249720 21666 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:00.281042 21666 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.135:41019
I20240501 20:09:00.281111 21792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.135:41019 every 8 connection(s)
I20240501 20:09:00.282330 21666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:09:00.289670 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 21666
I20240501 20:09:00.289861 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:09:00.293090 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.136:0
--local_ip_for_outbound_sockets=127.13.251.136
--webserver_interface=127.13.251.136
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:00.296011 21793 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:09:00.296352 21793 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:00.297313 21793 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:09:00.299018 20826 ts_manager.cc:194] Registered new tserver with Master: 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:00.299978 20826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.135:60567
I20240501 20:09:00.372551 21533 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
W20240501 20:09:00.633051 21796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:00.633339 21796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:00.633476 21796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:00.640966 21796 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:00.641105 21796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:00.641288 21796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.136
I20240501 20:09:00.650153 21796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.13.251.136
--webserver_port=0
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.136
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:00.651576 21796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:00.653388 21796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:00.665452 21801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:00.666240 21802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:00.667037 21804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:00.667747 21796 server_base.cc:1034] running on GCE node
I20240501 20:09:00.668354 21796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:00.669301 21796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:00.670496 21796 hybrid_clock.cc:648] HybridClock initialized: now 1714594140670439 us; error 54 us; skew 500 ppm
I20240501 20:09:00.670790 21796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:00.672974 21796 webserver.cc:460] Webserver started at http://127.13.251.136:42049/ using document root <none> and password file <none>
I20240501 20:09:00.673683 21796 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:00.673795 21796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:00.674098 21796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:00.676579 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "8559229e044b4e1ea661b881b3f9af50"
format_stamp: "Formatted at 2024-05-01 20:09:00 on dist-test-slave-qswb"
I20240501 20:09:00.677289 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "8559229e044b4e1ea661b881b3f9af50"
format_stamp: "Formatted at 2024-05-01 20:09:00 on dist-test-slave-qswb"
I20240501 20:09:00.682336 21796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:09:00.685616 21810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:00.686975 21796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:00.687148 21796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8559229e044b4e1ea661b881b3f9af50"
format_stamp: "Formatted at 2024-05-01 20:09:00 on dist-test-slave-qswb"
I20240501 20:09:00.687366 21796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:00.711719 21796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:00.712579 21796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:00.712833 21796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:00.713802 21796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:00.715392 21796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:00.715507 21796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:00.715638 21796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:00.715714 21796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:00.746807 21796 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.136:44267
I20240501 20:09:00.746891 21922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.136:44267 every 8 connection(s)
I20240501 20:09:00.748232 21796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:09:00.755838 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 21796
I20240501 20:09:00.756023 14318 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:09:00.760625 21923 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:09:00.760924 21923 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:00.761608 21923 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:09:00.762794 20826 ts_manager.cc:194] Registered new tserver with Master: 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:00.763593 20826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.136:37811
I20240501 20:09:00.772295 14318 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:09:00.789474 14318 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:09:00.789628 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 21406
I20240501 20:09:00.798240 21528 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:00.826859 21663 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
I20240501 20:09:00.879033 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 21406
I20240501 20:09:00.895895 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 21536
I20240501 20:09:00.904078 21658 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:00.986735 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 21536
I20240501 20:09:01.003692 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 21666
I20240501 20:09:01.011262 21788 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:01.090384 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 21666
I20240501 20:09:01.106580 14318 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6p4U5T/build/asan/bin/kudu pid 21796
I20240501 20:09:01.114953 21918 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:01.194558 14318 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6p4U5T/build/asan/bin/kudu with pid 21796
I20240501 20:09:04.964671 20826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:05.013649 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 67e11df69820456d8aa2ba056ca29a5d (DEFAULT_TABLE table=rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.017091 21078 tablet_service.cc:1467] Processing CreateTablet for tablet 398c2a957b7547ad9a92729242f5792f (DEFAULT_TABLE table=rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.017479 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 41558083a68b4ab09f4e762188537761 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.017717 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 67e11df69820456d8aa2ba056ca29a5d (DEFAULT_TABLE table=rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.018198 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 41558083a68b4ab09f4e762188537761 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.017632 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 398c2a957b7547ad9a92729242f5792f (DEFAULT_TABLE table=rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.018632 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398c2a957b7547ad9a92729242f5792f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.018998 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e11df69820456d8aa2ba056ca29a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.018929 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 41558083a68b4ab09f4e762188537761 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.019394 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41558083a68b4ab09f4e762188537761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.020677 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398c2a957b7547ad9a92729242f5792f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.021337 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41558083a68b4ab09f4e762188537761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.022130 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e11df69820456d8aa2ba056ca29a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.018589 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 67e11df69820456d8aa2ba056ca29a5d (DEFAULT_TABLE table=rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.023517 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 398c2a957b7547ad9a92729242f5792f (DEFAULT_TABLE table=rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.024747 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e11df69820456d8aa2ba056ca29a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.024837 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398c2a957b7547ad9a92729242f5792f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.029282 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41558083a68b4ab09f4e762188537761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.044603 21953 tablet_bootstrap.cc:492] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:05.049706 21954 tablet_bootstrap.cc:492] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:05.052235 21953 tablet_bootstrap.cc:654] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.053294 21954 tablet_bootstrap.cc:654] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.053889 21953 log.cc:824] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:05.054468 21954 log.cc:824] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:05.059082 21957 tablet_bootstrap.cc:492] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:05.063900 21953 tablet_bootstrap.cc:492] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:05.064276 21953 ts_tablet_manager.cc:1367] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20240501 20:09:05.064997 21952 tablet_bootstrap.cc:492] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:05.067979 21957 tablet_bootstrap.cc:654] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.069200 21952 tablet_bootstrap.cc:654] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.069272 21957 log.cc:824] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:05.069767 21954 tablet_bootstrap.cc:492] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:05.070063 21954 ts_tablet_manager.cc:1367] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.018s
I20240501 20:09:05.070681 21952 log.cc:824] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:05.075105 21953 raft_consensus.cc:348] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.075821 21953 raft_consensus.cc:374] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.075989 21953 raft_consensus.cc:729] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.077140 21953 consensus_queue.cc:260] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.080550 21957 tablet_bootstrap.cc:492] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:05.080854 21957 ts_tablet_manager.cc:1367] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.005s
I20240501 20:09:05.080657 21954 raft_consensus.cc:348] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.081096 21954 raft_consensus.cc:374] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.081223 21954 raft_consensus.cc:729] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.081406 21953 ts_tablet_manager.cc:1398] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20240501 20:09:05.081789 21953 tablet_bootstrap.cc:492] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:05.082006 21954 consensus_queue.cc:260] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.084589 21953 tablet_bootstrap.cc:654] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.089203 21952 tablet_bootstrap.cc:492] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:05.089668 21952 ts_tablet_manager.cc:1367] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.004s
I20240501 20:09:05.091825 21957 raft_consensus.cc:348] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.092437 21957 raft_consensus.cc:374] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.092567 21957 raft_consensus.cc:729] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.093477 21957 consensus_queue.cc:260] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.104319 21961 raft_consensus.cc:482] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:05.105208 21954 ts_tablet_manager.cc:1398] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.035s user 0.022s sys 0.004s
I20240501 20:09:05.105685 21954 tablet_bootstrap.cc:492] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:05.106541 21953 tablet_bootstrap.cc:492] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:05.105015 21961 raft_consensus.cc:504] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.106916 21953 ts_tablet_manager.cc:1367] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.016s
I20240501 20:09:05.106642 21952 raft_consensus.cc:348] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.107239 21952 raft_consensus.cc:374] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.107434 21954 tablet_bootstrap.cc:654] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.107374 21952 raft_consensus.cc:729] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.108719 21953 raft_consensus.cc:348] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:05.109035 21952 consensus_queue.cc:260] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.110937 21961 leader_election.cc:288] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:05.111166 21953 raft_consensus.cc:374] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.111346 21953 raft_consensus.cc:729] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.112569 21957 ts_tablet_manager.cc:1398] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.032s user 0.017s sys 0.012s
I20240501 20:09:05.113051 21957 tablet_bootstrap.cc:492] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:05.113248 21953 consensus_queue.cc:260] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:05.114059 21953 ts_tablet_manager.cc:1398] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 20:09:05.115886 21957 tablet_bootstrap.cc:654] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.116173 21953 tablet_bootstrap.cc:492] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:05.118464 21953 tablet_bootstrap.cc:654] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.126930 21954 tablet_bootstrap.cc:492] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:05.127157 21954 ts_tablet_manager.cc:1367] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240501 20:09:05.128072 21957 tablet_bootstrap.cc:492] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:05.128284 21957 ts_tablet_manager.cc:1367] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20240501 20:09:05.129062 21960 raft_consensus.cc:482] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:05.128607 21954 raft_consensus.cc:348] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.129216 21954 raft_consensus.cc:374] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.129339 21954 raft_consensus.cc:729] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.129303 21960 raft_consensus.cc:504] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.129401 21957 raft_consensus.cc:348] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:05.129598 21957 raft_consensus.cc:374] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.129678 21957 raft_consensus.cc:729] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.129671 21954 consensus_queue.cc:260] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.129904 21957 consensus_queue.cc:260] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:05.130470 21954 ts_tablet_manager.cc:1398] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:05.131404 21957 ts_tablet_manager.cc:1398] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:05.131433 21960 leader_election.cc:288] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:05.134974 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398c2a957b7547ad9a92729242f5792f" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:05.135638 21098 raft_consensus.cc:2454] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:05.137219 21291 leader_election.cc:302] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:05.137908 21952 ts_tablet_manager.cc:1398] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.048s user 0.025s sys 0.007s
I20240501 20:09:05.138342 21952 tablet_bootstrap.cc:492] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:05.139070 21963 raft_consensus.cc:2781] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:05.139238 21963 raft_consensus.cc:482] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:05.139364 21963 raft_consensus.cc:3037] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.140661 21952 tablet_bootstrap.cc:654] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.143374 21953 tablet_bootstrap.cc:492] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:05.143643 21953 ts_tablet_manager.cc:1367] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.008s
I20240501 20:09:05.143774 21963 raft_consensus.cc:504] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.144726 21953 raft_consensus.cc:348] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.145066 21953 raft_consensus.cc:374] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.145171 21953 raft_consensus.cc:729] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.145102 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398c2a957b7547ad9a92729242f5792f" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:05.145340 21098 raft_consensus.cc:3037] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.145465 21953 consensus_queue.cc:260] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.148983 21953 ts_tablet_manager.cc:1398] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:05.149389 21098 raft_consensus.cc:2454] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
W20240501 20:09:05.149405 21404 tablet.cc:2369] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:05.149855 21291 leader_election.cc:302] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:05.150313 21961 raft_consensus.cc:2781] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:05.151285 21963 leader_election.cc:288] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:05.157279 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398c2a957b7547ad9a92729242f5792f" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:05.157259 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398c2a957b7547ad9a92729242f5792f" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:05.157840 20967 raft_consensus.cc:2454] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:05.159870 21961 raft_consensus.cc:686] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:05.160349 21961 consensus_queue.cc:237] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.164872 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e11df69820456d8aa2ba056ca29a5d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:05.165426 21358 raft_consensus.cc:2454] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:05.166111 20901 leader_election.cc:302] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:05.166548 21960 raft_consensus.cc:2781] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:05.166698 21960 raft_consensus.cc:482] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:05.166812 21960 raft_consensus.cc:3037] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.169430 21960 raft_consensus.cc:504] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.170184 21960 leader_election.cc:288] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:05.170959 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e11df69820456d8aa2ba056ca29a5d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:05.171185 21358 raft_consensus.cc:3037] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.171379 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e11df69820456d8aa2ba056ca29a5d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:05.171841 21952 tablet_bootstrap.cc:492] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:05.171844 21227 raft_consensus.cc:3037] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.172021 21952 ts_tablet_manager.cc:1367] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.015s
I20240501 20:09:05.173032 21952 raft_consensus.cc:348] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:05.173261 21952 raft_consensus.cc:374] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.173345 21952 raft_consensus.cc:729] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.173563 21952 consensus_queue.cc:260] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:05.173950 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e11df69820456d8aa2ba056ca29a5d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:05.174118 21952 ts_tablet_manager.cc:1398] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:05.174261 21358 raft_consensus.cc:2454] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:05.174985 21227 raft_consensus.cc:2454] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:05.175577 20901 leader_election.cc:302] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:05.175943 21960 raft_consensus.cc:2781] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:05.175637 20826 catalog_manager.cc:5484] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:05.177033 21960 raft_consensus.cc:686] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:05.177697 21960 consensus_queue.cc:237] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:05.181771 20826 catalog_manager.cc:5484] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
W20240501 20:09:05.198400 21274 tablet.cc:2369] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:09:05.228157 21144 tablet.cc:2369] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:05.247020 21960 raft_consensus.cc:482] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:05.247295 21960 raft_consensus.cc:504] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:05.248088 21960 leader_election.cc:288] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:05.248593 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41558083a68b4ab09f4e762188537761" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:05.248720 21965 raft_consensus.cc:482] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:05.248880 21227 raft_consensus.cc:2454] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:05.248912 21965 raft_consensus.cc:504] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:05.249446 20903 leader_election.cc:302] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:05.249748 21960 raft_consensus.cc:2781] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:05.249842 21960 raft_consensus.cc:482] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:05.249962 21960 raft_consensus.cc:3037] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.251046 21965 leader_election.cc:288] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:05.252897 21960 raft_consensus.cc:504] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:05.253787 21960 leader_election.cc:288] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:05.254345 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41558083a68b4ab09f4e762188537761" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:05.254748 21227 raft_consensus.cc:3037] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.256891 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41558083a68b4ab09f4e762188537761" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:05.256951 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41558083a68b4ab09f4e762188537761" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:05.257174 21098 raft_consensus.cc:2454] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:05.258695 21227 raft_consensus.cc:2454] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:05.259289 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41558083a68b4ab09f4e762188537761" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:05.259387 20903 leader_election.cc:302] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters: feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:09:05.259578 21227 raft_consensus.cc:2379] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feac73db0a3b4aafa374ca5dfd322b08 in current term 1: Already voted for candidate cda19d6a81c0473db302535ce2ada7f0 in this term.
I20240501 20:09:05.259856 21960 raft_consensus.cc:2781] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:05.260056 21960 raft_consensus.cc:686] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:05.260342 21960 consensus_queue.cc:237] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
W20240501 20:09:05.262358 21013 tablet.cc:2369] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:05.263172 20826 catalog_manager.cc:5484] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:05.265123 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41558083a68b4ab09f4e762188537761" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:05.265707 21031 leader_election.cc:302] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feac73db0a3b4aafa374ca5dfd322b08; no voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0
I20240501 20:09:05.266091 21965 raft_consensus.cc:3037] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.268162 21965 raft_consensus.cc:2726] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:05.556995 20966 raft_consensus.cc:3037] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.561478 21961 consensus_queue.cc:1035] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.569042 21961 consensus_queue.cc:1035] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:05.642076 21981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:05.642418 21981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:05.650269 21981 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:05.665361 21960 consensus_queue.cc:1035] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.667989 21960 consensus_queue.cc:1035] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.672856 21976 consensus_queue.cc:1035] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.676664 21976 consensus_queue.cc:1035] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 254.4 ms
time per row: 0.1272 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:07.010560 20826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:07.019151 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 420b46edf5fc438fb01e822440bce8f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.019294 21078 tablet_service.cc:1467] Processing CreateTablet for tablet 420b46edf5fc438fb01e822440bce8f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.019695 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420b46edf5fc438fb01e822440bce8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.019742 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420b46edf5fc438fb01e822440bce8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.019990 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 420b46edf5fc438fb01e822440bce8f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.020375 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420b46edf5fc438fb01e822440bce8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.020777 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 24cc6ff1a0104ac690c8d8f82c17200c (DEFAULT_TABLE table=rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.021132 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc6ff1a0104ac690c8d8f82c17200c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.024668 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 24cc6ff1a0104ac690c8d8f82c17200c (DEFAULT_TABLE table=rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.025063 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc6ff1a0104ac690c8d8f82c17200c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.025722 21206 tablet_service.cc:1467] Processing CreateTablet for tablet f392e77df7ff4da489b4d8a8dc415af0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.026149 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f392e77df7ff4da489b4d8a8dc415af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.032094 22049 tablet_bootstrap.cc:492] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:07.032668 20947 tablet_service.cc:1467] Processing CreateTablet for tablet f392e77df7ff4da489b4d8a8dc415af0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.032864 22048 tablet_bootstrap.cc:492] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:07.033108 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f392e77df7ff4da489b4d8a8dc415af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.033500 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 24cc6ff1a0104ac690c8d8f82c17200c (DEFAULT_TABLE table=rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.033874 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc6ff1a0104ac690c8d8f82c17200c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.033949 22049 tablet_bootstrap.cc:654] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.034641 22048 tablet_bootstrap.cc:654] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.038183 21076 tablet_service.cc:1467] Processing CreateTablet for tablet f392e77df7ff4da489b4d8a8dc415af0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.038538 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f392e77df7ff4da489b4d8a8dc415af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.043605 22048 tablet_bootstrap.cc:492] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:07.043777 22048 ts_tablet_manager.cc:1367] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240501 20:09:07.044612 22048 raft_consensus.cc:348] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:07.044827 22048 raft_consensus.cc:374] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.044929 22048 raft_consensus.cc:729] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.045169 22048 consensus_queue.cc:260] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:07.045816 22048 ts_tablet_manager.cc:1398] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:07.046083 22048 tablet_bootstrap.cc:492] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:07.047735 22048 tablet_bootstrap.cc:654] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.049796 22049 tablet_bootstrap.cc:492] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:07.050065 22049 ts_tablet_manager.cc:1367] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20240501 20:09:07.051168 22049 raft_consensus.cc:348] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:07.051573 22049 raft_consensus.cc:374] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.051738 22049 raft_consensus.cc:729] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.052099 22049 consensus_queue.cc:260] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:07.053402 22053 tablet_bootstrap.cc:492] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:07.056268 22053 tablet_bootstrap.cc:654] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.056941 22049 ts_tablet_manager.cc:1398] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20240501 20:09:07.057349 22049 tablet_bootstrap.cc:492] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:07.059471 22049 tablet_bootstrap.cc:654] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.064777 22049 tablet_bootstrap.cc:492] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:07.064988 22049 ts_tablet_manager.cc:1367] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 20:09:07.065425 22053 tablet_bootstrap.cc:492] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:07.065613 22053 ts_tablet_manager.cc:1367] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240501 20:09:07.065807 22049 raft_consensus.cc:348] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.066056 22049 raft_consensus.cc:374] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.066171 22049 raft_consensus.cc:729] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.066428 22049 consensus_queue.cc:260] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.066780 22053 raft_consensus.cc:348] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.066951 22049 ts_tablet_manager.cc:1398] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:07.067075 22053 raft_consensus.cc:374] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.067234 22053 raft_consensus.cc:729] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.067523 22053 consensus_queue.cc:260] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.068167 22049 tablet_bootstrap.cc:492] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:07.070250 22049 tablet_bootstrap.cc:654] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.074335 22052 tablet_bootstrap.cc:492] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:07.074824 22053 ts_tablet_manager.cc:1398] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20240501 20:09:07.075313 22053 tablet_bootstrap.cc:492] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:07.076718 22052 tablet_bootstrap.cc:654] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.079684 22053 tablet_bootstrap.cc:654] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.080979 22048 tablet_bootstrap.cc:492] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:07.081317 22048 ts_tablet_manager.cc:1367] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.030s
I20240501 20:09:07.082365 22048 raft_consensus.cc:348] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.082661 22048 raft_consensus.cc:374] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.082743 22048 raft_consensus.cc:729] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.083004 22048 consensus_queue.cc:260] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.083209 22049 tablet_bootstrap.cc:492] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:07.083263 22053 tablet_bootstrap.cc:492] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:07.083370 22049 ts_tablet_manager.cc:1367] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20240501 20:09:07.083401 22053 ts_tablet_manager.cc:1367] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240501 20:09:07.083503 22048 ts_tablet_manager.cc:1398] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:07.084131 22049 raft_consensus.cc:348] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.084110 22053 raft_consensus.cc:348] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:07.084321 22049 raft_consensus.cc:374] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.084339 22053 raft_consensus.cc:374] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.084409 22053 raft_consensus.cc:729] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.084409 22049 raft_consensus.cc:729] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.084616 22052 tablet_bootstrap.cc:492] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:07.084630 22049 consensus_queue.cc:260] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.084657 22053 consensus_queue.cc:260] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:07.085175 22049 ts_tablet_manager.cc:1398] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:07.085203 22052 ts_tablet_manager.cc:1367] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240501 20:09:07.085245 22053 ts_tablet_manager.cc:1398] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:07.085522 22053 tablet_bootstrap.cc:492] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:07.086555 22052 raft_consensus.cc:348] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.086791 22052 raft_consensus.cc:374] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.087029 22053 tablet_bootstrap.cc:654] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.087122 22052 raft_consensus.cc:729] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.087378 22052 consensus_queue.cc:260] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.088693 22052 ts_tablet_manager.cc:1398] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:07.089560 22053 tablet_bootstrap.cc:492] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:07.089695 22053 ts_tablet_manager.cc:1367] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:07.090377 22053 raft_consensus.cc:348] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.090608 22053 raft_consensus.cc:374] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.090713 22053 raft_consensus.cc:729] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.090988 22053 consensus_queue.cc:260] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.091686 22053 ts_tablet_manager.cc:1398] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:07.112071 22055 raft_consensus.cc:482] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:07.112286 22055 raft_consensus.cc:504] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:07.113878 22055 leader_election.cc:288] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:07.135174 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420b46edf5fc438fb01e822440bce8f1" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:07.135428 21098 raft_consensus.cc:2454] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:07.135800 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420b46edf5fc438fb01e822440bce8f1" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:07.136035 21357 raft_consensus.cc:2454] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:07.136078 21161 leader_election.cc:302] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:07.136534 22055 raft_consensus.cc:2781] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:07.136662 22055 raft_consensus.cc:482] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:07.136760 22055 raft_consensus.cc:3037] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.138734 22055 raft_consensus.cc:504] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:07.139322 22055 leader_election.cc:288] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:07.139825 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420b46edf5fc438fb01e822440bce8f1" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:07.139883 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420b46edf5fc438fb01e822440bce8f1" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:07.140031 21098 raft_consensus.cc:3037] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.140053 21357 raft_consensus.cc:3037] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.141870 21098 raft_consensus.cc:2454] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:07.141878 21357 raft_consensus.cc:2454] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:07.142287 21162 leader_election.cc:302] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:07.142550 22055 raft_consensus.cc:2781] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:07.142971 22055 raft_consensus.cc:686] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:07.143464 22055 consensus_queue.cc:237] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:07.146919 20823 catalog_manager.cc:5484] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:07.205741 22055 raft_consensus.cc:482] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:07.205965 22055 raft_consensus.cc:504] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.206589 22055 leader_election.cc:288] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:07.207100 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc6ff1a0104ac690c8d8f82c17200c" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:07.207190 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc6ff1a0104ac690c8d8f82c17200c" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:07.207367 21357 raft_consensus.cc:2454] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:07.207422 21098 raft_consensus.cc:2454] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:07.207787 21162 leader_election.cc:302] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:07.208065 22055 raft_consensus.cc:2781] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:07.208163 22055 raft_consensus.cc:482] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:07.208245 22055 raft_consensus.cc:3037] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.210419 22055 raft_consensus.cc:504] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.210987 22055 leader_election.cc:288] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:07.211354 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc6ff1a0104ac690c8d8f82c17200c" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:07.211480 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc6ff1a0104ac690c8d8f82c17200c" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:07.211609 21357 raft_consensus.cc:3037] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.211716 21098 raft_consensus.cc:3037] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.214459 21357 raft_consensus.cc:2454] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:07.214475 21098 raft_consensus.cc:2454] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:07.214877 21162 leader_election.cc:302] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:07.215142 22055 raft_consensus.cc:2781] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:07.215319 22055 raft_consensus.cc:686] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:07.215581 22055 consensus_queue.cc:237] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.218199 20823 catalog_manager.cc:5484] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:07.237221 22058 raft_consensus.cc:482] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:07.237416 22058 raft_consensus.cc:504] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.238087 22058 leader_election.cc:288] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:07.238660 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f392e77df7ff4da489b4d8a8dc415af0" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:07.238659 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f392e77df7ff4da489b4d8a8dc415af0" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:07.238927 21228 raft_consensus.cc:2454] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:07.238925 20966 raft_consensus.cc:2454] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:07.239418 21031 leader_election.cc:302] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:07.239722 22058 raft_consensus.cc:2781] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:07.239835 22058 raft_consensus.cc:482] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:07.239928 22058 raft_consensus.cc:3037] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.241906 22058 raft_consensus.cc:504] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.242514 22058 leader_election.cc:288] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:07.242969 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f392e77df7ff4da489b4d8a8dc415af0" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:07.243016 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f392e77df7ff4da489b4d8a8dc415af0" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:07.243217 20966 raft_consensus.cc:3037] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.243224 21228 raft_consensus.cc:3037] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.246078 20966 raft_consensus.cc:2454] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:07.246078 21228 raft_consensus.cc:2454] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:07.246541 21034 leader_election.cc:302] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:07.246860 22058 raft_consensus.cc:2781] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:07.247380 22058 raft_consensus.cc:686] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:07.247921 22058 consensus_queue.cc:237] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:07.251072 20823 catalog_manager.cc:5484] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:07.575871 22055 consensus_queue.cc:1035] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.583402 22067 consensus_queue.cc:1035] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:07.634941 22072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:07.635288 22072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:07.642751 22072 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:09:07.686318 22067 consensus_queue.cc:1035] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.691416 22076 consensus_queue.cc:1035] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.764817 21228 raft_consensus.cc:1261] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:07.765383 22058 consensus_queue.cc:1035] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.769773 20966 raft_consensus.cc:1261] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:07.839610 22058 consensus_queue.cc:1035] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.856096 22112 mvcc.cc:205] Tried to move back new op lower bound from 7022977629237903360 to 7022977627131580416. Current Snapshot: MvccSnapshot[applied={T|T < 7022977629237903360}]
W20240501 20:09:07.899628 21031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 126685us
I20240501 20:09:07.945040 22126 mvcc.cc:205] Tried to move back new op lower bound from 7022977629237903360 to 7022977627131580416. Current Snapshot: MvccSnapshot[applied={T|T < 7022977629237903360}]
INSERT report
rows total: 2000
time total: 261.948 ms
time per row: 0.130974 ms
I20240501 20:09:08.071167 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:08.080060 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 80890a0703c2467fbdc2caaeec72ea7d (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.080327 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 80890a0703c2467fbdc2caaeec72ea7d (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.080590 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80890a0703c2467fbdc2caaeec72ea7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.080734 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80890a0703c2467fbdc2caaeec72ea7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.080788 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 80890a0703c2467fbdc2caaeec72ea7d (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.081193 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80890a0703c2467fbdc2caaeec72ea7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.082127 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 15e23b998afc4b0da79c87476bd55af6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.082528 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15e23b998afc4b0da79c87476bd55af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.084642 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 15e23b998afc4b0da79c87476bd55af6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.084890 21336 tablet_service.cc:1467] Processing CreateTablet for tablet f4cb707f4780467dbe97fc03978004fe (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.085058 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15e23b998afc4b0da79c87476bd55af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.085314 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cb707f4780467dbe97fc03978004fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.085680 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 15e23b998afc4b0da79c87476bd55af6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.086046 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15e23b998afc4b0da79c87476bd55af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.091912 20945 tablet_service.cc:1467] Processing CreateTablet for tablet f4cb707f4780467dbe97fc03978004fe (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.092276 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cb707f4780467dbe97fc03978004fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.096698 21078 tablet_service.cc:1467] Processing CreateTablet for tablet f4cb707f4780467dbe97fc03978004fe (DEFAULT_TABLE table=rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.097242 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4cb707f4780467dbe97fc03978004fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.097999 22136 tablet_bootstrap.cc:492] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:08.099474 22137 tablet_bootstrap.cc:492] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:08.100482 22136 tablet_bootstrap.cc:654] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.101100 22137 tablet_bootstrap.cc:654] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.103350 22137 tablet_bootstrap.cc:492] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:08.103482 22137 ts_tablet_manager.cc:1367] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:08.104279 22137 raft_consensus.cc:348] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:08.104532 22137 raft_consensus.cc:374] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.104640 22137 raft_consensus.cc:729] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.104890 22137 consensus_queue.cc:260] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:08.105511 22137 ts_tablet_manager.cc:1398] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:08.107260 22136 tablet_bootstrap.cc:492] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:08.107470 22136 ts_tablet_manager.cc:1367] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240501 20:09:08.108386 22136 raft_consensus.cc:348] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:08.108593 22136 raft_consensus.cc:374] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.108685 22136 raft_consensus.cc:729] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.108963 22136 consensus_queue.cc:260] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:08.109531 22136 ts_tablet_manager.cc:1398] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:08.115374 22140 tablet_bootstrap.cc:492] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:08.117142 22140 tablet_bootstrap.cc:654] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.118412 22137 tablet_bootstrap.cc:492] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:08.119997 22137 tablet_bootstrap.cc:654] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.121205 22135 tablet_bootstrap.cc:492] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:08.121330 22140 tablet_bootstrap.cc:492] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:08.121477 22140 ts_tablet_manager.cc:1367] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 20:09:08.122452 22140 raft_consensus.cc:348] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:08.122696 22135 tablet_bootstrap.cc:654] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.122735 22140 raft_consensus.cc:374] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.122854 22140 raft_consensus.cc:729] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.123175 22140 consensus_queue.cc:260] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:08.123966 22140 ts_tablet_manager.cc:1398] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:09:08.124248 22140 tablet_bootstrap.cc:492] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:08.124568 22135 tablet_bootstrap.cc:492] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:08.124722 22135 ts_tablet_manager.cc:1367] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:08.125322 22135 raft_consensus.cc:348] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:08.125496 22135 raft_consensus.cc:374] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.125586 22135 raft_consensus.cc:729] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.125782 22140 tablet_bootstrap.cc:654] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.125808 22135 consensus_queue.cc:260] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:08.126356 22135 ts_tablet_manager.cc:1398] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:08.126626 22135 tablet_bootstrap.cc:492] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:08.127864 22135 tablet_bootstrap.cc:654] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.131428 22137 tablet_bootstrap.cc:492] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:08.131650 22137 ts_tablet_manager.cc:1367] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20240501 20:09:08.132660 22137 raft_consensus.cc:348] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:08.132936 22137 raft_consensus.cc:374] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.133033 22137 raft_consensus.cc:729] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.133365 22137 consensus_queue.cc:260] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:08.136173 22137 ts_tablet_manager.cc:1398] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:08.137897 22137 tablet_bootstrap.cc:492] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:08.140408 22137 tablet_bootstrap.cc:654] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.142689 22137 tablet_bootstrap.cc:492] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:08.142802 22137 ts_tablet_manager.cc:1367] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:09:08.142949 22135 tablet_bootstrap.cc:492] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:08.143102 22135 ts_tablet_manager.cc:1367] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20240501 20:09:08.143416 22137 raft_consensus.cc:348] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:08.143599 22137 raft_consensus.cc:374] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.143687 22137 raft_consensus.cc:729] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.143748 22135 raft_consensus.cc:348] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:08.143939 22135 raft_consensus.cc:374] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.143901 22137 consensus_queue.cc:260] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:08.144012 22135 raft_consensus.cc:729] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.144248 22135 consensus_queue.cc:260] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:08.144542 22137 ts_tablet_manager.cc:1398] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:08.144728 22135 ts_tablet_manager.cc:1398] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:08.145013 22135 tablet_bootstrap.cc:492] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:08.145493 22140 tablet_bootstrap.cc:492] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:08.145697 22140 ts_tablet_manager.cc:1367] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20240501 20:09:08.146559 22135 tablet_bootstrap.cc:654] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.146476 22140 raft_consensus.cc:348] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:08.146703 22140 raft_consensus.cc:374] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.146775 22140 raft_consensus.cc:729] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.147042 22140 consensus_queue.cc:260] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:08.147688 22140 ts_tablet_manager.cc:1398] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:08.151466 22135 tablet_bootstrap.cc:492] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:08.151633 22135 ts_tablet_manager.cc:1367] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240501 20:09:08.152248 22135 raft_consensus.cc:348] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:08.152403 22135 raft_consensus.cc:374] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.152498 22135 raft_consensus.cc:729] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.152720 22135 consensus_queue.cc:260] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:08.153218 22135 ts_tablet_manager.cc:1398] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:08.169067 21960 raft_consensus.cc:482] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:08.169252 21960 raft_consensus.cc:504] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:08.169976 21960 leader_election.cc:288] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:08.170440 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e23b998afc4b0da79c87476bd55af6" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:08.170459 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e23b998afc4b0da79c87476bd55af6" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:08.170670 21228 raft_consensus.cc:2454] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:08.170696 21358 raft_consensus.cc:2454] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:08.171141 20903 leader_election.cc:302] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:08.171428 21960 raft_consensus.cc:2781] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:08.171522 21960 raft_consensus.cc:482] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:08.171631 21960 raft_consensus.cc:3037] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.173703 21960 raft_consensus.cc:504] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:08.174223 21960 leader_election.cc:288] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:08.174705 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e23b998afc4b0da79c87476bd55af6" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:08.174726 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15e23b998afc4b0da79c87476bd55af6" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:08.174877 21228 raft_consensus.cc:3037] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.174969 21358 raft_consensus.cc:3037] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.176802 21228 raft_consensus.cc:2454] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:08.177062 21358 raft_consensus.cc:2454] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:08.177201 20903 leader_election.cc:302] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:08.177525 21960 raft_consensus.cc:2781] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:08.177969 21960 raft_consensus.cc:686] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:08.178315 21960 consensus_queue.cc:237] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:08.181039 20823 catalog_manager.cc:5484] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:08.215596 22058 raft_consensus.cc:482] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:08.215832 22058 raft_consensus.cc:504] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:08.216557 22058 leader_election.cc:288] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:08.217249 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cb707f4780467dbe97fc03978004fe" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:08.217518 20966 raft_consensus.cc:2454] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:08.217983 21031 leader_election.cc:302] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:08.218286 22058 raft_consensus.cc:2781] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:08.218374 22058 raft_consensus.cc:482] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:08.218487 22058 raft_consensus.cc:3037] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.220503 22058 raft_consensus.cc:504] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:08.221069 22058 leader_election.cc:288] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:08.221562 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cb707f4780467dbe97fc03978004fe" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:08.221815 20966 raft_consensus.cc:3037] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.224624 20966 raft_consensus.cc:2454] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:08.224889 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cb707f4780467dbe97fc03978004fe" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:08.225066 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cb707f4780467dbe97fc03978004fe" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:08.225159 21031 leader_election.cc:302] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:08.225126 21358 raft_consensus.cc:2454] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:08.225912 22058 raft_consensus.cc:2781] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:08.226207 22058 raft_consensus.cc:686] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:08.226461 22058 consensus_queue.cc:237] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:08.230110 20823 catalog_manager.cc:5484] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:08.541955 21960 raft_consensus.cc:482] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:08.542156 21960 raft_consensus.cc:504] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:08.542809 21960 leader_election.cc:288] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:08.543361 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80890a0703c2467fbdc2caaeec72ea7d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:08.543385 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80890a0703c2467fbdc2caaeec72ea7d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:08.543597 21358 raft_consensus.cc:2454] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:08.543619 21097 raft_consensus.cc:2454] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:08.544040 20901 leader_election.cc:302] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:08.544368 21960 raft_consensus.cc:2781] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:08.544462 21960 raft_consensus.cc:482] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:08.544589 21960 raft_consensus.cc:3037] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.546341 21960 raft_consensus.cc:504] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:08.546947 21960 leader_election.cc:288] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:08.547392 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80890a0703c2467fbdc2caaeec72ea7d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:08.547389 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80890a0703c2467fbdc2caaeec72ea7d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:08.547621 21358 raft_consensus.cc:3037] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.547641 21097 raft_consensus.cc:3037] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.550321 21097 raft_consensus.cc:2454] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:08.550343 21358 raft_consensus.cc:2454] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:08.550767 20900 leader_election.cc:302] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:08.551064 21960 raft_consensus.cc:2781] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:08.551230 21960 raft_consensus.cc:686] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:08.551466 21960 consensus_queue.cc:237] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:08.553946 20823 catalog_manager.cc:5484] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:08.579667 21960 consensus_queue.cc:1035] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.587215 22148 consensus_queue.cc:1035] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.695274 22058 consensus_queue.cc:1035] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.699846 21358 raft_consensus.cc:3037] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.702482 22115 consensus_queue.cc:1035] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:08.910262 22154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:08.910665 22154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:08.922916 22154 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:08.943207 22148 consensus_queue.cc:1035] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.950253 21960 consensus_queue.cc:1035] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 212.683 ms
time per row: 0.106341 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:10.222612 20822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:10.233768 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 021fed14d5a140f3b42d359bae264f69 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.234200 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021fed14d5a140f3b42d359bae264f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.234131 21207 tablet_service.cc:1467] Processing CreateTablet for tablet aebb2a23076449fd9db492ed3e3b0cbf (DEFAULT_TABLE table=rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.234488 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebb2a23076449fd9db492ed3e3b0cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.234853 21338 tablet_service.cc:1467] Processing CreateTablet for tablet aebb2a23076449fd9db492ed3e3b0cbf (DEFAULT_TABLE table=rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.235237 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebb2a23076449fd9db492ed3e3b0cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.238098 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 021fed14d5a140f3b42d359bae264f69 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.238512 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021fed14d5a140f3b42d359bae264f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.240365 21078 tablet_service.cc:1467] Processing CreateTablet for tablet 021fed14d5a140f3b42d359bae264f69 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.240747 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021fed14d5a140f3b42d359bae264f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.241457 22220 tablet_bootstrap.cc:492] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:10.243534 22220 tablet_bootstrap.cc:654] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.243839 20946 tablet_service.cc:1467] Processing CreateTablet for tablet aebb2a23076449fd9db492ed3e3b0cbf (DEFAULT_TABLE table=rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.244127 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebb2a23076449fd9db492ed3e3b0cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.246098 22220 tablet_bootstrap.cc:492] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:10.246276 22220 ts_tablet_manager.cc:1367] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:09:10.247251 22220 raft_consensus.cc:348] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:10.247514 22220 raft_consensus.cc:374] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.247642 22220 raft_consensus.cc:729] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.248041 22220 consensus_queue.cc:260] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:10.248775 22220 ts_tablet_manager.cc:1398] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:10.253297 21077 tablet_service.cc:1467] Processing CreateTablet for tablet c1e295a46a174f3f8673e03f57890b58 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.253727 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e295a46a174f3f8673e03f57890b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.253715 21206 tablet_service.cc:1467] Processing CreateTablet for tablet c1e295a46a174f3f8673e03f57890b58 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.254078 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e295a46a174f3f8673e03f57890b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.257740 21337 tablet_service.cc:1467] Processing CreateTablet for tablet c1e295a46a174f3f8673e03f57890b58 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.258222 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e295a46a174f3f8673e03f57890b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.261698 22220 tablet_bootstrap.cc:492] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:10.263983 22220 tablet_bootstrap.cc:654] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.268854 22220 tablet_bootstrap.cc:492] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:10.269583 22220 ts_tablet_manager.cc:1367] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:09:10.270210 22222 tablet_bootstrap.cc:492] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:10.271530 22220 raft_consensus.cc:348] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:10.273507 22222 tablet_bootstrap.cc:654] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.273667 22220 raft_consensus.cc:374] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.273823 22220 raft_consensus.cc:729] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.274152 22220 consensus_queue.cc:260] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:10.274936 22220 ts_tablet_manager.cc:1398] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:10.275262 22220 tablet_bootstrap.cc:492] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:10.276548 22223 tablet_bootstrap.cc:492] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:10.276659 22219 tablet_bootstrap.cc:492] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:10.278002 22220 tablet_bootstrap.cc:654] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.278380 22219 tablet_bootstrap.cc:654] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.279259 22223 tablet_bootstrap.cc:654] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.281831 22219 tablet_bootstrap.cc:492] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:10.281896 22222 tablet_bootstrap.cc:492] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:10.281982 22219 ts_tablet_manager.cc:1367] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 20:09:10.282094 22222 ts_tablet_manager.cc:1367] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 20:09:10.282663 22219 raft_consensus.cc:348] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:10.282825 22219 raft_consensus.cc:374] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.282969 22219 raft_consensus.cc:729] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.282984 22222 raft_consensus.cc:348] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:10.283229 22222 raft_consensus.cc:374] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.283329 22222 raft_consensus.cc:729] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.283208 22219 consensus_queue.cc:260] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:10.283629 22222 consensus_queue.cc:260] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:10.283785 22223 tablet_bootstrap.cc:492] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:10.283794 22219 ts_tablet_manager.cc:1398] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:10.283923 22223 ts_tablet_manager.cc:1367] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240501 20:09:10.284118 22219 tablet_bootstrap.cc:492] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:10.284364 22222 ts_tablet_manager.cc:1398] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:09:10.284685 22222 tablet_bootstrap.cc:492] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:10.284732 22223 raft_consensus.cc:348] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:10.284981 22223 raft_consensus.cc:374] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.285130 22223 raft_consensus.cc:729] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.285435 22223 consensus_queue.cc:260] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:10.285682 22219 tablet_bootstrap.cc:654] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.286127 22223 ts_tablet_manager.cc:1398] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:10.286300 22222 tablet_bootstrap.cc:654] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.288239 22222 tablet_bootstrap.cc:492] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:10.288362 22223 tablet_bootstrap.cc:492] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:10.288446 22222 ts_tablet_manager.cc:1367] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 20:09:10.289168 22222 raft_consensus.cc:348] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:10.289435 22222 raft_consensus.cc:374] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.289583 22222 raft_consensus.cc:729] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.289917 22222 consensus_queue.cc:260] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:10.290620 22222 ts_tablet_manager.cc:1398] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:10.291357 22223 tablet_bootstrap.cc:654] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.293720 22220 tablet_bootstrap.cc:492] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:10.293996 22220 ts_tablet_manager.cc:1367] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240501 20:09:10.294806 22220 raft_consensus.cc:348] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:10.295091 22220 raft_consensus.cc:374] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.295202 22223 tablet_bootstrap.cc:492] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:10.295203 22220 raft_consensus.cc:729] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.295341 22219 tablet_bootstrap.cc:492] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:10.295351 22223 ts_tablet_manager.cc:1367] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 20:09:10.295508 22219 ts_tablet_manager.cc:1367] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20240501 20:09:10.295425 22220 consensus_queue.cc:260] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:10.296010 22220 ts_tablet_manager.cc:1398] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:10.296187 22219 raft_consensus.cc:348] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:10.296360 22219 raft_consensus.cc:374] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.296423 22219 raft_consensus.cc:729] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.296591 22223 raft_consensus.cc:348] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:10.296663 22219 consensus_queue.cc:260] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:10.296808 22223 raft_consensus.cc:374] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.296881 22223 raft_consensus.cc:729] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.297253 22219 ts_tablet_manager.cc:1398] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:10.297226 22223 consensus_queue.cc:260] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:10.297794 22223 ts_tablet_manager.cc:1398] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:10.306322 22076 raft_consensus.cc:482] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:10.306453 22043 raft_consensus.cc:482] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:10.306510 22076 raft_consensus.cc:504] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:10.306608 22043 raft_consensus.cc:504] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:10.307245 22076 leader_election.cc:288] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:10.307293 22043 leader_election.cc:288] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:10.308037 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebb2a23076449fd9db492ed3e3b0cbf" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:10.308331 20966 raft_consensus.cc:2454] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:10.308895 21291 leader_election.cc:302] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:10.309441 22043 raft_consensus.cc:2781] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:10.309594 22043 raft_consensus.cc:482] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:10.309697 22043 raft_consensus.cc:3037] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.313824 22043 raft_consensus.cc:504] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:10.316175 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebb2a23076449fd9db492ed3e3b0cbf" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:10.316509 20966 raft_consensus.cc:3037] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.320538 20966 raft_consensus.cc:2454] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:10.320978 22043 leader_election.cc:288] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:10.321560 21291 leader_election.cc:302] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:10.321897 22043 raft_consensus.cc:2781] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:10.325439 22043 raft_consensus.cc:686] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:10.325788 22043 consensus_queue.cc:237] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:10.326731 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021fed14d5a140f3b42d359bae264f69" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:10.326956 21097 raft_consensus.cc:2454] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:10.327510 21161 leader_election.cc:302] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:10.327937 22076 raft_consensus.cc:2781] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:10.328061 22076 raft_consensus.cc:482] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:10.328176 22076 raft_consensus.cc:3037] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.330228 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebb2a23076449fd9db492ed3e3b0cbf" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:10.330441 21227 raft_consensus.cc:3037] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.331007 22076 raft_consensus.cc:504] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:10.331447 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebb2a23076449fd9db492ed3e3b0cbf" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:10.331976 22076 leader_election.cc:288] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:10.332677 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021fed14d5a140f3b42d359bae264f69" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:10.332880 21097 raft_consensus.cc:3037] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.333403 21227 raft_consensus.cc:2454] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:10.334506 20823 catalog_manager.cc:5484] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:10.335088 21097 raft_consensus.cc:2454] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:10.335737 21161 leader_election.cc:302] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:10.336043 22076 raft_consensus.cc:2781] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:10.336429 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021fed14d5a140f3b42d359bae264f69" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:10.336555 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021fed14d5a140f3b42d359bae264f69" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:10.336771 20966 raft_consensus.cc:2454] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:10.336604 22076 raft_consensus.cc:686] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:10.337641 22076 consensus_queue.cc:237] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:10.340195 20823 catalog_manager.cc:5484] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:10.482268 22076 raft_consensus.cc:482] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:10.482442 22076 raft_consensus.cc:504] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:10.483018 22076 leader_election.cc:288] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:10.483439 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e295a46a174f3f8673e03f57890b58" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:10.483493 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e295a46a174f3f8673e03f57890b58" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:10.483659 21358 raft_consensus.cc:2454] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:10.483784 21097 raft_consensus.cc:2454] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:10.484074 21162 leader_election.cc:302] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:10.484325 22076 raft_consensus.cc:2781] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:10.484418 22076 raft_consensus.cc:482] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:10.484483 22076 raft_consensus.cc:3037] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.486351 22076 raft_consensus.cc:504] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:10.486981 22076 leader_election.cc:288] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:10.487351 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e295a46a174f3f8673e03f57890b58" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:10.487474 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e295a46a174f3f8673e03f57890b58" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:10.487569 21358 raft_consensus.cc:3037] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.487712 21097 raft_consensus.cc:3037] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.489444 21358 raft_consensus.cc:2454] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:10.489502 21097 raft_consensus.cc:2454] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:10.489842 21162 leader_election.cc:302] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:10.490134 22076 raft_consensus.cc:2781] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:10.490286 22076 raft_consensus.cc:686] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:10.490631 22076 consensus_queue.cc:237] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:10.493108 20823 catalog_manager.cc:5484] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:10.756246 20967 raft_consensus.cc:3037] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.758983 22076 consensus_queue.cc:1035] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.765316 22076 consensus_queue.cc:1035] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.857082 22043 consensus_queue.cc:1035] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.864219 22043 consensus_queue.cc:1035] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:10.879276 22240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:10.879650 22240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:10.887158 22240 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 20:09:11.071278 21098 raft_consensus.cc:1261] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:11.072507 22214 consensus_queue.cc:1035] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:11.087627 21358 raft_consensus.cc:1261] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:11.090445 22214 consensus_queue.cc:1035] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:11.100031 22248 mvcc.cc:205] Tried to move back new op lower bound from 7022977642778873856 to 7022977640411643904. Current Snapshot: MvccSnapshot[applied={T|T < 7022977642778873856}]
I20240501 20:09:11.123716 22245 mvcc.cc:205] Tried to move back new op lower bound from 7022977642778873856 to 7022977640411643904. Current Snapshot: MvccSnapshot[applied={T|T < 7022977642778873856}]
INSERT report
rows total: 2000
time total: 214.541 ms
time per row: 0.107271 ms
I20240501 20:09:11.255352 20822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:11.264326 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 832b77fbbad74123a5c6548732ed43a2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.264833 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832b77fbbad74123a5c6548732ed43a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.264649 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 832b77fbbad74123a5c6548732ed43a2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.265303 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832b77fbbad74123a5c6548732ed43a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.265257 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 90c5be8fb35f4298be590564144c5c22 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.265353 21078 tablet_service.cc:1467] Processing CreateTablet for tablet 832b77fbbad74123a5c6548732ed43a2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.265651 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c5be8fb35f4298be590564144c5c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.265749 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832b77fbbad74123a5c6548732ed43a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.272326 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 036f254eaa524900b20c25865f43d401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.272742 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036f254eaa524900b20c25865f43d401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.273190 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 90c5be8fb35f4298be590564144c5c22 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.273670 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c5be8fb35f4298be590564144c5c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.276217 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 90c5be8fb35f4298be590564144c5c22 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.277613 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 036f254eaa524900b20c25865f43d401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.278112 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036f254eaa524900b20c25865f43d401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.280156 22304 tablet_bootstrap.cc:492] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:11.282114 22304 tablet_bootstrap.cc:654] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.283182 22305 tablet_bootstrap.cc:492] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:11.285660 22305 tablet_bootstrap.cc:654] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.294556 22305 tablet_bootstrap.cc:492] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:11.294948 22305 ts_tablet_manager.cc:1367] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240501 20:09:11.296125 22305 raft_consensus.cc:348] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:11.296587 22305 raft_consensus.cc:374] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.296752 22305 raft_consensus.cc:729] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.297199 22305 consensus_queue.cc:260] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:11.298230 22308 tablet_bootstrap.cc:492] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:11.299072 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c5be8fb35f4298be590564144c5c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.301056 22303 tablet_bootstrap.cc:492] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:11.300025 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 036f254eaa524900b20c25865f43d401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.302062 22305 ts_tablet_manager.cc:1398] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:09:11.302069 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036f254eaa524900b20c25865f43d401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.303050 22308 tablet_bootstrap.cc:654] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.303511 22303 tablet_bootstrap.cc:654] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.310205 22308 tablet_bootstrap.cc:492] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:11.310837 22308 ts_tablet_manager.cc:1367] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20240501 20:09:11.311638 22308 raft_consensus.cc:348] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:11.311898 22305 tablet_bootstrap.cc:492] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:11.313040 22308 raft_consensus.cc:374] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.313139 22308 raft_consensus.cc:729] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.313704 22308 consensus_queue.cc:260] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:11.314107 22305 tablet_bootstrap.cc:654] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.314564 22308 ts_tablet_manager.cc:1398] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:11.314798 22308 tablet_bootstrap.cc:492] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:11.315810 22304 tablet_bootstrap.cc:492] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:11.315984 22304 ts_tablet_manager.cc:1367] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.015s
I20240501 20:09:11.316850 22304 raft_consensus.cc:348] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:11.317095 22304 raft_consensus.cc:374] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.317181 22304 raft_consensus.cc:729] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.317509 22303 tablet_bootstrap.cc:492] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:11.317405 22304 consensus_queue.cc:260] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:11.317641 22303 ts_tablet_manager.cc:1367] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20240501 20:09:11.317999 22304 ts_tablet_manager.cc:1398] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:11.318250 22304 tablet_bootstrap.cc:492] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:11.318421 22303 raft_consensus.cc:348] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:11.318598 22303 raft_consensus.cc:374] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.318677 22303 raft_consensus.cc:729] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.318917 22303 consensus_queue.cc:260] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:11.319505 22303 ts_tablet_manager.cc:1398] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:11.319856 22304 tablet_bootstrap.cc:654] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.321123 22308 tablet_bootstrap.cc:654] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.322500 22303 tablet_bootstrap.cc:492] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:11.325263 22303 tablet_bootstrap.cc:654] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.327579 22305 tablet_bootstrap.cc:492] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:11.327824 22303 tablet_bootstrap.cc:492] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:11.327826 22304 tablet_bootstrap.cc:492] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:11.327869 22305 ts_tablet_manager.cc:1367] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.006s
I20240501 20:09:11.327983 22304 ts_tablet_manager.cc:1367] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:09:11.328051 22303 ts_tablet_manager.cc:1367] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 20:09:11.328411 22308 tablet_bootstrap.cc:492] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:11.328615 22308 ts_tablet_manager.cc:1367] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20240501 20:09:11.328784 22304 raft_consensus.cc:348] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:11.328769 22305 raft_consensus.cc:348] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:11.329013 22305 raft_consensus.cc:374] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.328845 22303 raft_consensus.cc:348] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:11.329056 22304 raft_consensus.cc:374] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.329164 22304 raft_consensus.cc:729] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.329167 22305 raft_consensus.cc:729] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.329183 22303 raft_consensus.cc:374] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.329289 22303 raft_consensus.cc:729] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.329490 22308 raft_consensus.cc:348] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:11.329469 22304 consensus_queue.cc:260] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:11.329496 22305 consensus_queue.cc:260] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:11.329597 22303 consensus_queue.cc:260] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:11.330001 22304 ts_tablet_manager.cc:1398] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:11.330149 22305 ts_tablet_manager.cc:1398] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:11.330651 22303 ts_tablet_manager.cc:1398] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:11.330789 22308 raft_consensus.cc:374] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.330893 22308 raft_consensus.cc:729] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.331116 22308 consensus_queue.cc:260] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:11.331636 22308 ts_tablet_manager.cc:1398] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:11.332018 22303 tablet_bootstrap.cc:492] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:11.333493 22303 tablet_bootstrap.cc:654] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.336773 22303 tablet_bootstrap.cc:492] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:11.336932 22303 ts_tablet_manager.cc:1367] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 20:09:11.337531 22303 raft_consensus.cc:348] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:11.337677 22303 raft_consensus.cc:374] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.337747 22303 raft_consensus.cc:729] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.337965 22303 consensus_queue.cc:260] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:11.338478 22303 ts_tablet_manager.cc:1398] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:11.378262 22174 raft_consensus.cc:482] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:11.378477 22174 raft_consensus.cc:504] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:11.379190 22174 leader_election.cc:288] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:11.379791 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036f254eaa524900b20c25865f43d401" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:11.379933 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036f254eaa524900b20c25865f43d401" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:11.380044 21358 raft_consensus.cc:2454] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:11.380199 21227 raft_consensus.cc:2454] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:11.380563 20901 leader_election.cc:302] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:11.380908 22174 raft_consensus.cc:2781] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:11.381035 22174 raft_consensus.cc:482] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:11.381121 22174 raft_consensus.cc:3037] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:11.383217 22174 raft_consensus.cc:504] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:11.383837 22174 leader_election.cc:288] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:11.384227 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036f254eaa524900b20c25865f43d401" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:11.384354 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036f254eaa524900b20c25865f43d401" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:11.384441 21358 raft_consensus.cc:3037] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:11.384552 21227 raft_consensus.cc:3037] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:11.387351 21227 raft_consensus.cc:2454] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:11.387367 21358 raft_consensus.cc:2454] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:11.387784 20901 leader_election.cc:302] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:11.388074 22174 raft_consensus.cc:2781] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:11.388456 22174 raft_consensus.cc:686] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:11.388700 22174 consensus_queue.cc:237] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:11.391355 20818 catalog_manager.cc:5484] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:11.434664 22174 raft_consensus.cc:482] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:11.434864 22174 raft_consensus.cc:504] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:11.435461 22174 leader_election.cc:288] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:11.436002 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832b77fbbad74123a5c6548732ed43a2" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:11.436050 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832b77fbbad74123a5c6548732ed43a2" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:11.436220 21098 raft_consensus.cc:2454] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:11.436348 21358 raft_consensus.cc:2454] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:11.436693 20900 leader_election.cc:302] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:11.436956 22174 raft_consensus.cc:2781] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:11.437062 22174 raft_consensus.cc:482] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:11.437170 22174 raft_consensus.cc:3037] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:11.439075 22174 raft_consensus.cc:504] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:11.439690 22174 leader_election.cc:288] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:11.440088 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832b77fbbad74123a5c6548732ed43a2" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:11.440294 21098 raft_consensus.cc:3037] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:11.440244 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832b77fbbad74123a5c6548732ed43a2" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:11.440447 21358 raft_consensus.cc:3037] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:11.442353 21358 raft_consensus.cc:2454] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:11.442725 20901 leader_election.cc:302] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:11.442971 22174 raft_consensus.cc:2781] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:11.443113 22174 raft_consensus.cc:686] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:11.443131 21098 raft_consensus.cc:2454] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:11.443377 22174 consensus_queue.cc:237] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:11.449220 20818 catalog_manager.cc:5484] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:11.503355 22058 raft_consensus.cc:482] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:11.503664 22058 raft_consensus.cc:504] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:11.504331 22058 leader_election.cc:288] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:11.504884 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c5be8fb35f4298be590564144c5c22" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:11.504941 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c5be8fb35f4298be590564144c5c22" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:11.505177 21227 raft_consensus.cc:2454] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:11.505192 20965 raft_consensus.cc:2454] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:11.505623 21034 leader_election.cc:302] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:11.505909 22058 raft_consensus.cc:2781] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:11.506000 22058 raft_consensus.cc:482] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:11.506101 22058 raft_consensus.cc:3037] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:11.507999 22058 raft_consensus.cc:504] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:11.508582 22058 leader_election.cc:288] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:11.508945 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c5be8fb35f4298be590564144c5c22" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:11.509167 21227 raft_consensus.cc:3037] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:11.509112 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c5be8fb35f4298be590564144c5c22" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:11.509322 20965 raft_consensus.cc:3037] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:11.512022 21227 raft_consensus.cc:2454] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:11.512367 20965 raft_consensus.cc:2454] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:11.512455 21034 leader_election.cc:302] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:11.512805 22058 raft_consensus.cc:2781] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:11.513296 22058 raft_consensus.cc:686] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:11.513588 22058 consensus_queue.cc:237] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:11.516414 20818 catalog_manager.cc:5484] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:11.858461 22174 consensus_queue.cc:1035] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:11.864578 22174 consensus_queue.cc:1035] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:11.911850 22321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:11.912277 22321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:11.924456 22321 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:11.947180 22174 consensus_queue.cc:1035] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:11.951992 22174 consensus_queue.cc:1035] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240501 20:09:12.032619 22058 consensus_queue.cc:1035] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:12.040388 22058 consensus_queue.cc:1035] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:12.212247 20901 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 48029.1us
W20240501 20:09:12.240969 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 84325.1us
INSERT report
rows total: 2000
time total: 283.443 ms
time per row: 0.141721 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:13.298476 20825 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:13.307258 21338 tablet_service.cc:1467] Processing CreateTablet for tablet d25f96315839453d8cd5458417404e34 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:13.307760 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f96315839453d8cd5458417404e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.308166 21206 tablet_service.cc:1467] Processing CreateTablet for tablet d25f96315839453d8cd5458417404e34 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:13.308544 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f96315839453d8cd5458417404e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.308713 20945 tablet_service.cc:1467] Processing CreateTablet for tablet e5da8abdb3014753ac3cbef1d622069e (DEFAULT_TABLE table=rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:13.308796 21077 tablet_service.cc:1467] Processing CreateTablet for tablet d25f96315839453d8cd5458417404e34 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:13.309132 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5da8abdb3014753ac3cbef1d622069e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.309152 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f96315839453d8cd5458417404e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.312306 21076 tablet_service.cc:1467] Processing CreateTablet for tablet e5da8abdb3014753ac3cbef1d622069e (DEFAULT_TABLE table=rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:13.312650 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5da8abdb3014753ac3cbef1d622069e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.314029 21078 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4384496d9048259bf353bd58a15008 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:13.314370 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4384496d9048259bf353bd58a15008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.315119 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4384496d9048259bf353bd58a15008 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:13.315491 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4384496d9048259bf353bd58a15008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.319896 21336 tablet_service.cc:1467] Processing CreateTablet for tablet e5da8abdb3014753ac3cbef1d622069e (DEFAULT_TABLE table=rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:13.320298 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5da8abdb3014753ac3cbef1d622069e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.324133 22391 tablet_bootstrap.cc:492] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:13.324776 22389 tablet_bootstrap.cc:492] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:13.324785 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4384496d9048259bf353bd58a15008 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:13.325218 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4384496d9048259bf353bd58a15008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.325784 22391 tablet_bootstrap.cc:654] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.326252 22389 tablet_bootstrap.cc:654] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.329586 22390 tablet_bootstrap.cc:492] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:13.332563 22390 tablet_bootstrap.cc:654] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.334453 22392 tablet_bootstrap.cc:492] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:13.337188 22392 tablet_bootstrap.cc:654] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.338263 22391 tablet_bootstrap.cc:492] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:13.338524 22391 ts_tablet_manager.cc:1367] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.013s
I20240501 20:09:13.340169 22391 raft_consensus.cc:348] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.341207 22391 raft_consensus.cc:374] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.341428 22391 raft_consensus.cc:729] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.341863 22391 consensus_queue.cc:260] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.344264 22391 ts_tablet_manager.cc:1398] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:13.344599 22391 tablet_bootstrap.cc:492] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:13.346393 22391 tablet_bootstrap.cc:654] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.352547 22389 tablet_bootstrap.cc:492] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:13.352797 22389 ts_tablet_manager.cc:1367] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.017s
I20240501 20:09:13.353017 22392 tablet_bootstrap.cc:492] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:13.353233 22392 ts_tablet_manager.cc:1367] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.004s
I20240501 20:09:13.353634 22389 raft_consensus.cc:348] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.353932 22389 raft_consensus.cc:374] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.354079 22389 raft_consensus.cc:729] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.354056 22392 raft_consensus.cc:348] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:13.354275 22390 tablet_bootstrap.cc:492] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:13.354306 22392 raft_consensus.cc:374] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.354465 22391 tablet_bootstrap.cc:492] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:13.354456 22390 ts_tablet_manager.cc:1367] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.007s
I20240501 20:09:13.354465 22392 raft_consensus.cc:729] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.354458 22389 consensus_queue.cc:260] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.354663 22391 ts_tablet_manager.cc:1367] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 20:09:13.354818 22392 consensus_queue.cc:260] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:13.355196 22390 raft_consensus.cc:348] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.355417 22390 raft_consensus.cc:374] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.355525 22390 raft_consensus.cc:729] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.355424 22391 raft_consensus.cc:348] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.355827 22390 consensus_queue.cc:260] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.356177 22389 ts_tablet_manager.cc:1398] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:13.356472 22389 tablet_bootstrap.cc:492] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:13.356508 22390 ts_tablet_manager.cc:1398] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:13.356798 22390 tablet_bootstrap.cc:492] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:13.357151 22392 ts_tablet_manager.cc:1398] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:13.358402 22389 tablet_bootstrap.cc:654] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.358532 22390 tablet_bootstrap.cc:654] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.355669 22391 raft_consensus.cc:374] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.363533 22391 raft_consensus.cc:729] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.364096 22391 consensus_queue.cc:260] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.365170 22389 tablet_bootstrap.cc:492] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:13.365371 22391 ts_tablet_manager.cc:1398] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:09:13.365384 22389 ts_tablet_manager.cc:1367] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240501 20:09:13.365548 22390 tablet_bootstrap.cc:492] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:13.365734 22391 tablet_bootstrap.cc:492] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:13.365780 22390 ts_tablet_manager.cc:1367] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.005s
I20240501 20:09:13.366204 22389 raft_consensus.cc:348] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:13.366463 22389 raft_consensus.cc:374] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.366590 22389 raft_consensus.cc:729] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.366830 22390 raft_consensus.cc:348] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.366885 22389 consensus_queue.cc:260] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:13.367203 22390 raft_consensus.cc:374] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.367333 22390 raft_consensus.cc:729] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.367527 22389 ts_tablet_manager.cc:1398] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:13.367676 22391 tablet_bootstrap.cc:654] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.367730 22390 consensus_queue.cc:260] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.367959 22389 tablet_bootstrap.cc:492] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:13.368288 22390 ts_tablet_manager.cc:1398] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:13.370364 22389 tablet_bootstrap.cc:654] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.370711 22391 tablet_bootstrap.cc:492] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:13.370887 22391 ts_tablet_manager.cc:1367] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 20:09:13.371717 22391 raft_consensus.cc:348] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:13.371970 22391 raft_consensus.cc:374] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.372097 22391 raft_consensus.cc:729] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.372364 22391 consensus_queue.cc:260] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:13.373064 22391 ts_tablet_manager.cc:1398] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:13.374253 22389 tablet_bootstrap.cc:492] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:13.374401 22389 ts_tablet_manager.cc:1367] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:09:13.374977 22389 raft_consensus.cc:348] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.375137 22389 raft_consensus.cc:374] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.375200 22389 raft_consensus.cc:729] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.375411 22389 consensus_queue.cc:260] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.375972 22389 ts_tablet_manager.cc:1398] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 20:09:13.618815 22353 raft_consensus.cc:482] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:13.619032 22353 raft_consensus.cc:504] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:13.619719 22353 leader_election.cc:288] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:13.620330 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5da8abdb3014753ac3cbef1d622069e" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:13.620323 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5da8abdb3014753ac3cbef1d622069e" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:13.620556 21358 raft_consensus.cc:2454] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:13.620579 20965 raft_consensus.cc:2454] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:13.621031 21031 leader_election.cc:302] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:13.621327 22353 raft_consensus.cc:2781] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:13.621456 22353 raft_consensus.cc:482] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:13.621546 22353 raft_consensus.cc:3037] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.623405 22353 raft_consensus.cc:504] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:13.624045 22353 leader_election.cc:288] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:13.624542 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5da8abdb3014753ac3cbef1d622069e" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:13.624581 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5da8abdb3014753ac3cbef1d622069e" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:13.624758 21358 raft_consensus.cc:3037] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.624769 20965 raft_consensus.cc:3037] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.627586 20965 raft_consensus.cc:2454] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:13.627599 21358 raft_consensus.cc:2454] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:13.628087 21032 leader_election.cc:302] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:13.628401 22353 raft_consensus.cc:2781] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:13.628751 22353 raft_consensus.cc:686] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:13.629069 22353 consensus_queue.cc:237] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:13.631727 20819 catalog_manager.cc:5484] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:13.641862 22353 raft_consensus.cc:482] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:13.642079 22353 raft_consensus.cc:504] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.642652 22353 leader_election.cc:288] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:13.643117 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f96315839453d8cd5458417404e34" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:13.643134 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f96315839453d8cd5458417404e34" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:13.643353 21358 raft_consensus.cc:2454] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:13.643378 21228 raft_consensus.cc:2454] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:13.643748 21032 leader_election.cc:302] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:13.644064 22353 raft_consensus.cc:2781] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:13.644163 22353 raft_consensus.cc:482] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:13.644279 22353 raft_consensus.cc:3037] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.681617 22353 raft_consensus.cc:504] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.682422 22353 leader_election.cc:288] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:13.683041 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f96315839453d8cd5458417404e34" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:13.683041 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f96315839453d8cd5458417404e34" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:13.683329 21358 raft_consensus.cc:3037] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.683324 21228 raft_consensus.cc:3037] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.686334 21358 raft_consensus.cc:2454] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:13.686334 21228 raft_consensus.cc:2454] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:13.686985 21032 leader_election.cc:302] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:13.687515 22353 raft_consensus.cc:2781] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:13.687780 22353 raft_consensus.cc:686] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:13.688133 22353 consensus_queue.cc:237] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.691468 20819 catalog_manager.cc:5484] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:13.731810 22247 raft_consensus.cc:482] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:13.732048 22247 raft_consensus.cc:504] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.732827 22247 leader_election.cc:288] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:13.733331 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4384496d9048259bf353bd58a15008" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:13.733448 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4384496d9048259bf353bd58a15008" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:13.733594 21097 raft_consensus.cc:2454] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:13.733697 21228 raft_consensus.cc:2454] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:13.734127 21291 leader_election.cc:302] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:13.734409 22247 raft_consensus.cc:2781] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:13.734551 22247 raft_consensus.cc:482] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:13.734650 22247 raft_consensus.cc:3037] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.736784 22247 raft_consensus.cc:504] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.737560 22247 leader_election.cc:288] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:13.737965 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4384496d9048259bf353bd58a15008" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:13.738178 21097 raft_consensus.cc:3037] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.738484 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4384496d9048259bf353bd58a15008" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:13.738682 21228 raft_consensus.cc:3037] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.740221 21097 raft_consensus.cc:2454] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:13.740617 21291 leader_election.cc:302] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:13.740926 22247 raft_consensus.cc:2781] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:13.740940 21228 raft_consensus.cc:2454] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:13.741431 22247 raft_consensus.cc:686] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:13.741742 22247 consensus_queue.cc:237] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:13.744596 20819 catalog_manager.cc:5484] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:14.069096 22353 consensus_queue.cc:1035] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:14.075423 22353 consensus_queue.cc:1035] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:14.108595 22402 consensus_queue.cc:1035] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:14.113656 22353 consensus_queue.cc:1035] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:14.133394 22408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:14.133744 22408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:14.141255 22408 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:14.145673 22247 consensus_queue.cc:1035] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:14.150022 22247 consensus_queue.cc:1035] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 220.569 ms
time per row: 0.110284 ms
I20240501 20:09:14.524376 20819 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:14.533510 21077 tablet_service.cc:1467] Processing CreateTablet for tablet be7bba5002cc4c72b46884ef394a3d15 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.533881 21206 tablet_service.cc:1467] Processing CreateTablet for tablet be7bba5002cc4c72b46884ef394a3d15 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.534029 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7bba5002cc4c72b46884ef394a3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.534261 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7bba5002cc4c72b46884ef394a3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.534624 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 9845d5032786442e8da848ebf3b111dd (DEFAULT_TABLE table=rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.535040 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9845d5032786442e8da848ebf3b111dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.535095 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 9845d5032786442e8da848ebf3b111dd (DEFAULT_TABLE table=rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.535455 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9845d5032786442e8da848ebf3b111dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.538679 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 493824843c024d50b1a4698ca13d2089 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.539083 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493824843c024d50b1a4698ca13d2089. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.539219 20945 tablet_service.cc:1467] Processing CreateTablet for tablet be7bba5002cc4c72b46884ef394a3d15 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.539770 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7bba5002cc4c72b46884ef394a3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.541632 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 493824843c024d50b1a4698ca13d2089 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.542011 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493824843c024d50b1a4698ca13d2089. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.542232 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 9845d5032786442e8da848ebf3b111dd (DEFAULT_TABLE table=rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.542568 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9845d5032786442e8da848ebf3b111dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.546438 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 493824843c024d50b1a4698ca13d2089 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.546885 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493824843c024d50b1a4698ca13d2089. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.551081 22473 tablet_bootstrap.cc:492] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:14.554219 22473 tablet_bootstrap.cc:654] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.554838 22472 tablet_bootstrap.cc:492] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:14.557260 22472 tablet_bootstrap.cc:654] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.558241 22473 tablet_bootstrap.cc:492] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:14.558382 22473 ts_tablet_manager.cc:1367] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:09:14.560422 22473 raft_consensus.cc:348] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:14.561178 22473 raft_consensus.cc:374] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.561318 22473 raft_consensus.cc:729] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.561914 22473 consensus_queue.cc:260] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:14.562872 22473 ts_tablet_manager.cc:1398] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:14.563853 22473 tablet_bootstrap.cc:492] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:14.565702 22473 tablet_bootstrap.cc:654] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.567126 22472 tablet_bootstrap.cc:492] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:14.567755 22472 ts_tablet_manager.cc:1367] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240501 20:09:14.568145 22474 tablet_bootstrap.cc:492] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:14.568866 22472 raft_consensus.cc:348] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:14.569180 22472 raft_consensus.cc:374] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.569321 22472 raft_consensus.cc:729] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.569641 22472 consensus_queue.cc:260] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:14.569986 22474 tablet_bootstrap.cc:654] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.570479 22472 ts_tablet_manager.cc:1398] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:14.570787 22472 tablet_bootstrap.cc:492] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:14.571413 22471 tablet_bootstrap.cc:492] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:14.572914 22472 tablet_bootstrap.cc:654] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.573228 22471 tablet_bootstrap.cc:654] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.575704 22473 tablet_bootstrap.cc:492] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:14.575887 22473 ts_tablet_manager.cc:1367] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 20:09:14.576396 22474 tablet_bootstrap.cc:492] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:14.576570 22474 ts_tablet_manager.cc:1367] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:09:14.576735 22473 raft_consensus.cc:348] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:14.576915 22473 raft_consensus.cc:374] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.576985 22473 raft_consensus.cc:729] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.577235 22473 consensus_queue.cc:260] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:14.577544 22474 raft_consensus.cc:348] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:14.577790 22473 ts_tablet_manager.cc:1398] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:14.577793 22474 raft_consensus.cc:374] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.577903 22474 raft_consensus.cc:729] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.578079 22473 tablet_bootstrap.cc:492] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:14.578207 22474 consensus_queue.cc:260] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:14.579723 22473 tablet_bootstrap.cc:654] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.580235 22474 ts_tablet_manager.cc:1398] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:14.580574 22474 tablet_bootstrap.cc:492] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:14.581629 22471 tablet_bootstrap.cc:492] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:14.581830 22471 ts_tablet_manager.cc:1367] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240501 20:09:14.582211 22474 tablet_bootstrap.cc:654] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.582806 22471 raft_consensus.cc:348] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:14.583073 22471 raft_consensus.cc:374] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.583250 22471 raft_consensus.cc:729] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.583670 22471 consensus_queue.cc:260] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:14.584681 22471 ts_tablet_manager.cc:1398] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:14.585959 22472 tablet_bootstrap.cc:492] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:14.585994 22474 tablet_bootstrap.cc:492] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:14.586153 22472 ts_tablet_manager.cc:1367] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20240501 20:09:14.586293 22474 ts_tablet_manager.cc:1367] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:09:14.586925 22472 raft_consensus.cc:348] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:14.587157 22472 raft_consensus.cc:374] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.587234 22472 raft_consensus.cc:729] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.587061 22474 raft_consensus.cc:348] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:14.587469 22474 raft_consensus.cc:374] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.587610 22474 raft_consensus.cc:729] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.587491 22472 consensus_queue.cc:260] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:14.588150 22472 ts_tablet_manager.cc:1398] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:14.588310 22474 consensus_queue.cc:260] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:14.589675 22472 tablet_bootstrap.cc:492] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:14.589779 22474 ts_tablet_manager.cc:1398] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:14.591655 22402 raft_consensus.cc:482] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:14.591789 22402 raft_consensus.cc:504] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:14.592051 22473 tablet_bootstrap.cc:492] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:14.592222 22472 tablet_bootstrap.cc:654] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.592222 22473 ts_tablet_manager.cc:1367] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240501 20:09:14.592481 22402 leader_election.cc:288] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:14.593005 22473 raft_consensus.cc:348] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:14.593153 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bba5002cc4c72b46884ef394a3d15" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:14.593257 22473 raft_consensus.cc:374] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.593370 22473 raft_consensus.cc:729] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.593374 20965 raft_consensus.cc:2454] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:14.593648 22473 consensus_queue.cc:260] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:14.593860 21031 leader_election.cc:302] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:14.594321 22473 ts_tablet_manager.cc:1398] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:14.594389 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bba5002cc4c72b46884ef394a3d15" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:14.594568 21226 raft_consensus.cc:2454] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:14.595348 22402 raft_consensus.cc:2781] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:14.595441 22402 raft_consensus.cc:482] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:14.595690 22402 raft_consensus.cc:3037] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.597028 22472 tablet_bootstrap.cc:492] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:14.597199 22472 ts_tablet_manager.cc:1367] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 20:09:14.598003 22472 raft_consensus.cc:348] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:14.598173 22472 raft_consensus.cc:374] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.598181 22402 raft_consensus.cc:504] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:14.598281 22472 raft_consensus.cc:729] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.598495 22472 consensus_queue.cc:260] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:14.599224 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bba5002cc4c72b46884ef394a3d15" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:14.599447 21226 raft_consensus.cc:3037] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.599650 22472 ts_tablet_manager.cc:1398] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:09:14.599733 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bba5002cc4c72b46884ef394a3d15" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:14.599928 20965 raft_consensus.cc:3037] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.600196 22402 leader_election.cc:288] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:14.602303 21226 raft_consensus.cc:2454] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:14.602619 20965 raft_consensus.cc:2454] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:14.602689 21034 leader_election.cc:302] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:14.602977 22402 raft_consensus.cc:2781] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:14.603238 22402 raft_consensus.cc:686] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:14.603516 22402 consensus_queue.cc:237] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:14.606146 20824 catalog_manager.cc:5484] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:14.662297 22402 raft_consensus.cc:482] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:14.662484 22402 raft_consensus.cc:504] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:14.663157 22402 leader_election.cc:288] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:14.663625 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493824843c024d50b1a4698ca13d2089" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:14.663792 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493824843c024d50b1a4698ca13d2089" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:14.663902 21226 raft_consensus.cc:2454] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:14.664036 20965 raft_consensus.cc:2454] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:14.664422 21034 leader_election.cc:302] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:14.664820 22402 raft_consensus.cc:2781] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:14.664923 22402 raft_consensus.cc:482] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:14.665006 22402 raft_consensus.cc:3037] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.666870 22402 raft_consensus.cc:504] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:14.667416 22402 leader_election.cc:288] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:14.667856 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493824843c024d50b1a4698ca13d2089" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:14.668045 21226 raft_consensus.cc:3037] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.668051 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493824843c024d50b1a4698ca13d2089" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:14.668285 20965 raft_consensus.cc:3037] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.669930 21226 raft_consensus.cc:2454] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:14.670285 20965 raft_consensus.cc:2454] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:14.670312 21034 leader_election.cc:302] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:14.670607 22402 raft_consensus.cc:2781] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:14.670765 22402 raft_consensus.cc:686] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:14.671038 22402 consensus_queue.cc:237] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:14.674089 20824 catalog_manager.cc:5484] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:14.770256 22442 raft_consensus.cc:482] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:14.770438 22442 raft_consensus.cc:504] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:14.771100 22442 leader_election.cc:288] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:14.771589 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9845d5032786442e8da848ebf3b111dd" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:14.771685 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9845d5032786442e8da848ebf3b111dd" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:14.771858 21358 raft_consensus.cc:2454] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:14.771912 21226 raft_consensus.cc:2454] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:14.772320 20901 leader_election.cc:302] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:14.772619 22442 raft_consensus.cc:2781] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:14.772750 22442 raft_consensus.cc:482] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:14.772840 22442 raft_consensus.cc:3037] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.774659 22442 raft_consensus.cc:504] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:14.775184 22442 leader_election.cc:288] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:14.775684 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9845d5032786442e8da848ebf3b111dd" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:14.775708 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9845d5032786442e8da848ebf3b111dd" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:14.775848 21226 raft_consensus.cc:3037] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.775878 21358 raft_consensus.cc:3037] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.777758 21358 raft_consensus.cc:2454] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:14.777812 21226 raft_consensus.cc:2454] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:14.778201 20901 leader_election.cc:302] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:14.778514 22442 raft_consensus.cc:2781] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:14.778918 22442 raft_consensus.cc:686] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:14.779253 22442 consensus_queue.cc:237] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:14.782217 20824 catalog_manager.cc:5484] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:15.023362 22353 consensus_queue.cc:1035] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:15.029901 22402 consensus_queue.cc:1035] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:15.150312 22488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:15.150635 22488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:15.158653 22488 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:15.177139 22442 consensus_queue.cc:1035] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:15.183018 22486 consensus_queue.cc:1035] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240501 20:09:15.206014 22402 consensus_queue.cc:1035] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:15.211977 22402 consensus_queue.cc:1035] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 193.098 ms
time per row: 0.0965489 ms
I20240501 20:09:15.503362 20819 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:15.512626 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 554b1d57dbb541508afaf8dbb77cbdad (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.513118 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554b1d57dbb541508afaf8dbb77cbdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.513720 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 72726d6b786e4c72ace7359101cec949 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.513784 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 554b1d57dbb541508afaf8dbb77cbdad (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.513984 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 10a4d24c5c844c8c9ccd191b1a2b0b00 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.514214 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72726d6b786e4c72ace7359101cec949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.514426 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a4d24c5c844c8c9ccd191b1a2b0b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.515120 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 10a4d24c5c844c8c9ccd191b1a2b0b00 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.515261 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 554b1d57dbb541508afaf8dbb77cbdad (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.515480 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a4d24c5c844c8c9ccd191b1a2b0b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.515666 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554b1d57dbb541508afaf8dbb77cbdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.519804 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554b1d57dbb541508afaf8dbb77cbdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.519837 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 72726d6b786e4c72ace7359101cec949 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.520220 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72726d6b786e4c72ace7359101cec949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.524015 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 10a4d24c5c844c8c9ccd191b1a2b0b00 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.524441 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a4d24c5c844c8c9ccd191b1a2b0b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.528003 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 72726d6b786e4c72ace7359101cec949 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.528476 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72726d6b786e4c72ace7359101cec949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.534597 22553 tablet_bootstrap.cc:492] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:15.537791 22553 tablet_bootstrap.cc:654] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.538640 22551 tablet_bootstrap.cc:492] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:15.541373 22551 tablet_bootstrap.cc:654] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.542552 22555 tablet_bootstrap.cc:492] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:15.546404 22555 tablet_bootstrap.cc:654] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.547299 22552 tablet_bootstrap.cc:492] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:15.549520 22551 tablet_bootstrap.cc:492] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:15.548724 22553 tablet_bootstrap.cc:492] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:15.549774 22551 ts_tablet_manager.cc:1367] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 20:09:15.550022 22553 ts_tablet_manager.cc:1367] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240501 20:09:15.550249 22552 tablet_bootstrap.cc:654] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.550801 22551 raft_consensus.cc:348] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:15.551213 22551 raft_consensus.cc:374] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.551187 22553 raft_consensus.cc:348] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:15.552048 22553 raft_consensus.cc:374] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.552338 22553 raft_consensus.cc:729] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.551383 22551 raft_consensus.cc:729] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.552825 22553 consensus_queue.cc:260] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:15.552909 22551 consensus_queue.cc:260] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:15.553757 22551 ts_tablet_manager.cc:1398] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:15.554638 22551 tablet_bootstrap.cc:492] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:15.556052 22555 tablet_bootstrap.cc:492] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:15.556255 22555 ts_tablet_manager.cc:1367] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20240501 20:09:15.556964 22555 raft_consensus.cc:348] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:15.557190 22555 raft_consensus.cc:374] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.557317 22555 raft_consensus.cc:729] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.557564 22555 consensus_queue.cc:260] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:15.558635 22552 tablet_bootstrap.cc:492] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:15.558861 22552 ts_tablet_manager.cc:1367] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 20:09:15.559024 22555 ts_tablet_manager.cc:1398] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 20:09:15.559315 22551 tablet_bootstrap.cc:654] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.559336 22555 tablet_bootstrap.cc:492] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:15.559903 22552 raft_consensus.cc:348] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:15.560140 22552 raft_consensus.cc:374] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.560317 22552 raft_consensus.cc:729] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.560655 22552 consensus_queue.cc:260] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:15.560958 22555 tablet_bootstrap.cc:654] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.561458 22552 ts_tablet_manager.cc:1398] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:15.561856 22552 tablet_bootstrap.cc:492] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:15.563740 22552 tablet_bootstrap.cc:654] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.565328 22551 tablet_bootstrap.cc:492] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:15.566140 22551 ts_tablet_manager.cc:1367] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20240501 20:09:15.567337 22551 raft_consensus.cc:348] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:15.567763 22551 raft_consensus.cc:374] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.567915 22551 raft_consensus.cc:729] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.568341 22551 consensus_queue.cc:260] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:15.569281 22551 ts_tablet_manager.cc:1398] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:15.572252 22552 tablet_bootstrap.cc:492] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:15.572479 22552 ts_tablet_manager.cc:1367] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20240501 20:09:15.573346 22552 raft_consensus.cc:348] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:15.573674 22552 raft_consensus.cc:374] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.573834 22552 raft_consensus.cc:729] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.574165 22552 consensus_queue.cc:260] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:15.574916 22552 ts_tablet_manager.cc:1398] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:15.578104 22553 ts_tablet_manager.cc:1398] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240501 20:09:15.578583 22553 tablet_bootstrap.cc:492] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:15.581786 22553 tablet_bootstrap.cc:654] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.583669 22555 tablet_bootstrap.cc:492] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:15.583845 22553 tablet_bootstrap.cc:492] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:15.583855 22555 ts_tablet_manager.cc:1367] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.007s
I20240501 20:09:15.584010 22553 ts_tablet_manager.cc:1367] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:09:15.584599 22555 raft_consensus.cc:348] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:15.584830 22555 raft_consensus.cc:374] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.584962 22555 raft_consensus.cc:729] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.584883 22553 raft_consensus.cc:348] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:15.585165 22553 raft_consensus.cc:374] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.585193 22555 consensus_queue.cc:260] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:15.585281 22553 raft_consensus.cc:729] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.585538 22553 consensus_queue.cc:260] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:15.586197 22553 ts_tablet_manager.cc:1398] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:15.586648 22553 tablet_bootstrap.cc:492] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:15.588057 22555 ts_tablet_manager.cc:1398] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:15.589529 22553 tablet_bootstrap.cc:654] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.592016 22553 tablet_bootstrap.cc:492] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:15.592157 22553 ts_tablet_manager.cc:1367] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:09:15.592999 22553 raft_consensus.cc:348] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:15.593215 22553 raft_consensus.cc:374] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.593319 22553 raft_consensus.cc:729] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.593555 22553 consensus_queue.cc:260] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:15.594213 22553 ts_tablet_manager.cc:1398] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:15.597980 22247 raft_consensus.cc:482] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:15.598145 22247 raft_consensus.cc:504] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:15.598649 22247 leader_election.cc:288] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:15.599081 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554b1d57dbb541508afaf8dbb77cbdad" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:15.599161 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554b1d57dbb541508afaf8dbb77cbdad" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:15.599280 20967 raft_consensus.cc:2454] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:15.599422 21097 raft_consensus.cc:2454] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:15.599705 21291 leader_election.cc:302] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:15.600005 22247 raft_consensus.cc:2781] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:15.600092 22247 raft_consensus.cc:482] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:15.600206 22247 raft_consensus.cc:3037] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.602047 22247 raft_consensus.cc:504] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:15.602528 22247 leader_election.cc:288] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:15.602984 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554b1d57dbb541508afaf8dbb77cbdad" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:15.603183 20967 raft_consensus.cc:3037] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.603396 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554b1d57dbb541508afaf8dbb77cbdad" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:15.603596 21097 raft_consensus.cc:3037] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.605140 20967 raft_consensus.cc:2454] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:15.605470 21291 leader_election.cc:302] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:15.605521 21097 raft_consensus.cc:2454] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:15.605789 22247 raft_consensus.cc:2781] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:15.606230 22247 raft_consensus.cc:686] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:15.606504 22247 consensus_queue.cc:237] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:15.609380 20821 catalog_manager.cc:5484] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:15.785248 22247 raft_consensus.cc:482] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:15.785418 22247 raft_consensus.cc:504] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:15.786010 22247 leader_election.cc:288] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:15.786448 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72726d6b786e4c72ace7359101cec949" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:15.786489 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72726d6b786e4c72ace7359101cec949" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:15.786641 21097 raft_consensus.cc:2454] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:15.786693 21228 raft_consensus.cc:2454] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:15.787036 21291 leader_election.cc:302] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:15.787302 22247 raft_consensus.cc:2781] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:15.787392 22247 raft_consensus.cc:482] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:15.787451 22247 raft_consensus.cc:3037] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.789228 22247 raft_consensus.cc:504] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:15.789665 22247 leader_election.cc:288] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:15.790076 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72726d6b786e4c72ace7359101cec949" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:15.790112 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72726d6b786e4c72ace7359101cec949" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:15.790249 21097 raft_consensus.cc:3037] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.790347 21228 raft_consensus.cc:3037] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.792256 21228 raft_consensus.cc:2454] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:15.792701 21097 raft_consensus.cc:2454] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:15.792686 21294 leader_election.cc:302] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:15.793020 22247 raft_consensus.cc:2781] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:15.793216 22247 raft_consensus.cc:686] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:15.793457 22247 consensus_queue.cc:237] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:15.796241 20821 catalog_manager.cc:5484] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:15.890590 22290 raft_consensus.cc:482] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:15.890794 22290 raft_consensus.cc:504] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:15.891466 22290 leader_election.cc:288] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:15.891916 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4d24c5c844c8c9ccd191b1a2b0b00" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:15.892174 21358 raft_consensus.cc:2454] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:15.892228 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4d24c5c844c8c9ccd191b1a2b0b00" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:15.892480 20967 raft_consensus.cc:2454] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:15.892925 21162 leader_election.cc:302] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:15.893210 22290 raft_consensus.cc:2781] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:15.893337 22290 raft_consensus.cc:482] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:15.893430 22290 raft_consensus.cc:3037] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.896329 22290 raft_consensus.cc:504] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:15.896975 22290 leader_election.cc:288] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:15.897416 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4d24c5c844c8c9ccd191b1a2b0b00" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:15.897475 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4d24c5c844c8c9ccd191b1a2b0b00" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:15.897655 21358 raft_consensus.cc:3037] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.897677 20967 raft_consensus.cc:3037] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.900559 21358 raft_consensus.cc:2454] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:15.900552 20967 raft_consensus.cc:2454] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:15.901001 21162 leader_election.cc:302] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:15.901322 22290 raft_consensus.cc:2781] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:15.901684 22290 raft_consensus.cc:686] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:15.902307 22290 consensus_queue.cc:237] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:15.905401 20821 catalog_manager.cc:5484] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:16.170506 22247 consensus_queue.cc:1035] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:16.177083 22247 consensus_queue.cc:1035] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:16.224542 22564 consensus_queue.cc:1035] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:16.232401 22572 consensus_queue.cc:1035] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:16.271077 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:16.271421 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:16.279469 22569 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 20:09:16.404544 22290 consensus_queue.cc:1035] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:16.414911 22290 consensus_queue.cc:1035] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 239.704 ms
time per row: 0.119852 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:17.641225 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:17.651007 20946 tablet_service.cc:1467] Processing CreateTablet for tablet f552dfb425734b6f943914725274d1e4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:17.651487 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f552dfb425734b6f943914725274d1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.651445 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 85be3407c8724e7d89e1f4df16808e9a (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:17.651857 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85be3407c8724e7d89e1f4df16808e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.652160 21076 tablet_service.cc:1467] Processing CreateTablet for tablet f552dfb425734b6f943914725274d1e4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:17.652484 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f552dfb425734b6f943914725274d1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.656004 21208 tablet_service.cc:1467] Processing CreateTablet for tablet f999e1ac85544f228dad6e422352d759 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:17.656409 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f999e1ac85544f228dad6e422352d759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.657574 22638 tablet_bootstrap.cc:492] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:17.657951 21337 tablet_service.cc:1467] Processing CreateTablet for tablet f552dfb425734b6f943914725274d1e4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:17.658362 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f552dfb425734b6f943914725274d1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.660260 22638 tablet_bootstrap.cc:654] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.661958 21336 tablet_service.cc:1467] Processing CreateTablet for tablet f999e1ac85544f228dad6e422352d759 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:17.662482 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f999e1ac85544f228dad6e422352d759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.668944 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 85be3407c8724e7d89e1f4df16808e9a (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:17.668921 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 85be3407c8724e7d89e1f4df16808e9a (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:17.669376 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85be3407c8724e7d89e1f4df16808e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.669477 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85be3407c8724e7d89e1f4df16808e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.670006 22641 tablet_bootstrap.cc:492] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:17.672531 20945 tablet_service.cc:1467] Processing CreateTablet for tablet f999e1ac85544f228dad6e422352d759 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:17.673061 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f999e1ac85544f228dad6e422352d759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:17.673607 22641 tablet_bootstrap.cc:654] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.683385 22642 tablet_bootstrap.cc:492] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:17.683923 22638 tablet_bootstrap.cc:492] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:17.684197 22638 ts_tablet_manager.cc:1367] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.007s
I20240501 20:09:17.685220 22642 tablet_bootstrap.cc:654] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.685120 22638 raft_consensus.cc:348] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.685364 22638 raft_consensus.cc:374] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.685493 22638 raft_consensus.cc:729] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.685806 22638 consensus_queue.cc:260] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.686601 22638 ts_tablet_manager.cc:1398] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:17.689263 22639 tablet_bootstrap.cc:492] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:17.690583 22638 tablet_bootstrap.cc:492] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:17.691615 22641 tablet_bootstrap.cc:492] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:17.691812 22641 ts_tablet_manager.cc:1367] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.006s
I20240501 20:09:17.692629 22641 raft_consensus.cc:348] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.692936 22641 raft_consensus.cc:374] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.692957 22638 tablet_bootstrap.cc:654] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.693090 22641 raft_consensus.cc:729] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.693434 22642 tablet_bootstrap.cc:492] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:17.693450 22641 consensus_queue.cc:260] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.693625 22642 ts_tablet_manager.cc:1367] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:09:17.694361 22639 tablet_bootstrap.cc:654] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.694391 22642 raft_consensus.cc:348] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.694634 22642 raft_consensus.cc:374] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.694782 22642 raft_consensus.cc:729] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.695057 22642 consensus_queue.cc:260] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.696133 22642 ts_tablet_manager.cc:1398] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:17.696517 22642 tablet_bootstrap.cc:492] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:17.698096 22642 tablet_bootstrap.cc:654] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.699332 22641 ts_tablet_manager.cc:1398] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:09:17.701881 22641 tablet_bootstrap.cc:492] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:17.704612 22638 tablet_bootstrap.cc:492] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:17.704764 22641 tablet_bootstrap.cc:654] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.704878 22638 ts_tablet_manager.cc:1367] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240501 20:09:17.705495 22639 tablet_bootstrap.cc:492] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:17.705694 22639 ts_tablet_manager.cc:1367] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20240501 20:09:17.705746 22638 raft_consensus.cc:348] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:17.706060 22638 raft_consensus.cc:374] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.706207 22638 raft_consensus.cc:729] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.706509 22639 raft_consensus.cc:348] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:17.706538 22638 consensus_queue.cc:260] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:17.706719 22639 raft_consensus.cc:374] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.706802 22639 raft_consensus.cc:729] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.707083 22639 consensus_queue.cc:260] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:17.707428 22638 ts_tablet_manager.cc:1398] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:17.707695 22639 ts_tablet_manager.cc:1398] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:17.710266 22639 tablet_bootstrap.cc:492] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:17.712400 22639 tablet_bootstrap.cc:654] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.714716 22642 tablet_bootstrap.cc:492] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:17.714898 22641 tablet_bootstrap.cc:492] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:17.714942 22642 ts_tablet_manager.cc:1367] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20240501 20:09:17.715073 22641 ts_tablet_manager.cc:1367] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240501 20:09:17.715848 22641 raft_consensus.cc:348] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:17.715848 22642 raft_consensus.cc:348] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.716076 22642 raft_consensus.cc:374] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.716078 22641 raft_consensus.cc:374] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.716154 22641 raft_consensus.cc:729] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.716193 22642 raft_consensus.cc:729] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.716488 22642 consensus_queue.cc:260] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.716460 22641 consensus_queue.cc:260] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:17.717240 22641 ts_tablet_manager.cc:1398] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:17.717552 22641 tablet_bootstrap.cc:492] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:17.717973 22639 tablet_bootstrap.cc:492] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:17.718132 22639 ts_tablet_manager.cc:1367] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240501 20:09:17.718807 22639 raft_consensus.cc:348] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.719117 22639 raft_consensus.cc:374] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.719244 22639 raft_consensus.cc:729] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.719297 22641 tablet_bootstrap.cc:654] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:17.719581 22639 consensus_queue.cc:260] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.720269 22639 ts_tablet_manager.cc:1398] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:17.721361 22642 ts_tablet_manager.cc:1398] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:09:17.722787 22641 tablet_bootstrap.cc:492] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:17.723009 22641 ts_tablet_manager.cc:1367] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:09:17.723877 22641 raft_consensus.cc:348] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.724184 22641 raft_consensus.cc:374] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:17.724293 22641 raft_consensus.cc:729] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:17.724546 22641 consensus_queue.cc:260] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.724889 22632 raft_consensus.cc:482] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:17.725066 22632 raft_consensus.cc:504] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.725414 22641 ts_tablet_manager.cc:1398] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:17.725777 22632 leader_election.cc:288] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:17.726111 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f999e1ac85544f228dad6e422352d759" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:17.726333 21226 raft_consensus.cc:2454] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:17.726413 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f999e1ac85544f228dad6e422352d759" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:17.726624 21356 raft_consensus.cc:2454] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:17.726788 20903 leader_election.cc:302] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:17.727057 22632 raft_consensus.cc:2781] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:17.727180 22632 raft_consensus.cc:482] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:17.727263 22632 raft_consensus.cc:3037] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.729336 22632 raft_consensus.cc:504] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.729892 22632 leader_election.cc:288] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:17.730275 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f999e1ac85544f228dad6e422352d759" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:17.730356 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f999e1ac85544f228dad6e422352d759" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:17.730494 21226 raft_consensus.cc:3037] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.730542 21356 raft_consensus.cc:3037] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.733306 21226 raft_consensus.cc:2454] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:17.733321 21356 raft_consensus.cc:2454] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:17.733726 20901 leader_election.cc:302] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:17.734067 22632 raft_consensus.cc:2781] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:17.734202 22632 raft_consensus.cc:686] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:17.734452 22632 consensus_queue.cc:237] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.737295 20819 catalog_manager.cc:5484] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:17.906526 22632 raft_consensus.cc:482] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:17.906737 22632 raft_consensus.cc:504] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.907389 22632 leader_election.cc:288] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:17.907974 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f552dfb425734b6f943914725274d1e4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:17.908219 21098 raft_consensus.cc:2454] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:17.908845 20900 leader_election.cc:302] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:17.908962 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f552dfb425734b6f943914725274d1e4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:17.909202 22632 raft_consensus.cc:2781] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:17.909202 21356 raft_consensus.cc:2454] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:17.909288 22632 raft_consensus.cc:482] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:17.909377 22632 raft_consensus.cc:3037] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.910307 22353 raft_consensus.cc:482] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:17.910454 22353 raft_consensus.cc:504] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:17.911062 22353 leader_election.cc:288] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:17.911723 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85be3407c8724e7d89e1f4df16808e9a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:17.911967 20965 raft_consensus.cc:2454] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:17.911433 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85be3407c8724e7d89e1f4df16808e9a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:17.912048 22632 raft_consensus.cc:504] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.912256 21226 raft_consensus.cc:2454] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:17.912432 21031 leader_election.cc:302] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:17.912711 22632 leader_election.cc:288] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:17.912860 22654 raft_consensus.cc:2781] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:17.912961 22654 raft_consensus.cc:482] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:17.913141 22654 raft_consensus.cc:3037] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.913136 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f552dfb425734b6f943914725274d1e4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:17.913146 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f552dfb425734b6f943914725274d1e4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:17.913344 21356 raft_consensus.cc:3037] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.913403 21098 raft_consensus.cc:3037] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.915702 21098 raft_consensus.cc:2454] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:17.916064 20900 leader_election.cc:302] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:17.916055 21356 raft_consensus.cc:2454] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:17.916076 22654 raft_consensus.cc:504] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:17.916339 22632 raft_consensus.cc:2781] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:17.916497 22632 raft_consensus.cc:686] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:17.916800 22654 leader_election.cc:288] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:17.916831 22632 consensus_queue.cc:237] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:17.917176 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85be3407c8724e7d89e1f4df16808e9a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:17.917294 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85be3407c8724e7d89e1f4df16808e9a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:17.917380 21226 raft_consensus.cc:3037] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.917511 20965 raft_consensus.cc:3037] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:17.919981 21226 raft_consensus.cc:2454] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:17.920239 20965 raft_consensus.cc:2454] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:17.920447 21034 leader_election.cc:302] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:17.920466 20819 catalog_manager.cc:5484] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:17.920732 22654 raft_consensus.cc:2781] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:17.920866 22654 raft_consensus.cc:686] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:17.921111 22654 consensus_queue.cc:237] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:17.924808 20819 catalog_manager.cc:5484] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:18.264001 22632 consensus_queue.cc:1035] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:18.270835 22632 consensus_queue.cc:1035] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:18.329033 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:18.329411 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:18.336877 22657 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:18.366930 22651 consensus_queue.cc:1035] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240501 20:09:18.378484 22651 consensus_queue.cc:1035] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:18.432700 22654 consensus_queue.cc:1035] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:18.437884 22353 consensus_queue.cc:1035] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:18.648828 21031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 60835.6us
INSERT report
rows total: 2000
time total: 279.335 ms
time per row: 0.139667 ms
I20240501 20:09:18.769965 20819 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:18.778854 21337 tablet_service.cc:1467] Processing CreateTablet for tablet a115fe66d11143ea8e24aa891e4e2f33 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:18.779060 21077 tablet_service.cc:1467] Processing CreateTablet for tablet a115fe66d11143ea8e24aa891e4e2f33 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:18.779376 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a115fe66d11143ea8e24aa891e4e2f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:18.779506 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a115fe66d11143ea8e24aa891e4e2f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:18.779868 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 43e80df9af0e40c5acc4dea95d7e95a8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:18.780257 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e80df9af0e40c5acc4dea95d7e95a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:18.780640 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 43e80df9af0e40c5acc4dea95d7e95a8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:18.781000 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e80df9af0e40c5acc4dea95d7e95a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:18.784212 21078 tablet_service.cc:1467] Processing CreateTablet for tablet aa02aed57a7a4e6a95b149d2078dc265 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:18.784619 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa02aed57a7a4e6a95b149d2078dc265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:18.787006 21207 tablet_service.cc:1467] Processing CreateTablet for tablet a115fe66d11143ea8e24aa891e4e2f33 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:18.787425 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a115fe66d11143ea8e24aa891e4e2f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:18.791903 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 43e80df9af0e40c5acc4dea95d7e95a8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:18.792264 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e80df9af0e40c5acc4dea95d7e95a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:18.793852 22722 tablet_bootstrap.cc:492] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:18.795258 22723 tablet_bootstrap.cc:492] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:18.796530 20946 tablet_service.cc:1467] Processing CreateTablet for tablet aa02aed57a7a4e6a95b149d2078dc265 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:18.796747 21336 tablet_service.cc:1467] Processing CreateTablet for tablet aa02aed57a7a4e6a95b149d2078dc265 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:18.796980 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa02aed57a7a4e6a95b149d2078dc265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:18.797173 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa02aed57a7a4e6a95b149d2078dc265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:18.797780 22723 tablet_bootstrap.cc:654] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:18.806229 22724 tablet_bootstrap.cc:492] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:18.806427 22723 tablet_bootstrap.cc:492] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:18.806582 22723 ts_tablet_manager.cc:1367] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 20:09:18.807296 22723 raft_consensus.cc:348] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.807499 22723 raft_consensus.cc:374] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:18.807624 22724 tablet_bootstrap.cc:654] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:18.807646 22723 raft_consensus.cc:729] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:18.807906 22723 consensus_queue.cc:260] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.808413 22722 tablet_bootstrap.cc:654] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:18.808630 22723 ts_tablet_manager.cc:1398] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:18.808921 22723 tablet_bootstrap.cc:492] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:18.810345 22724 tablet_bootstrap.cc:492] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:18.810576 22724 ts_tablet_manager.cc:1367] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:09:18.810637 22723 tablet_bootstrap.cc:654] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:18.812986 22724 raft_consensus.cc:348] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.814288 22726 tablet_bootstrap.cc:492] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:18.813316 22724 raft_consensus.cc:374] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:18.815407 22724 raft_consensus.cc:729] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:18.815758 22724 consensus_queue.cc:260] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.816376 22724 ts_tablet_manager.cc:1398] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:18.816680 22726 tablet_bootstrap.cc:654] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:18.816833 22724 tablet_bootstrap.cc:492] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:18.819576 22724 tablet_bootstrap.cc:654] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:18.821067 22722 tablet_bootstrap.cc:492] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:18.821322 22722 ts_tablet_manager.cc:1367] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20240501 20:09:18.822697 22722 raft_consensus.cc:348] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.823096 22722 raft_consensus.cc:374] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:18.823252 22722 raft_consensus.cc:729] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:18.823670 22722 consensus_queue.cc:260] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.824535 22722 ts_tablet_manager.cc:1398] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:18.826741 22722 tablet_bootstrap.cc:492] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:18.828641 22722 tablet_bootstrap.cc:654] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:18.830024 22723 tablet_bootstrap.cc:492] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:18.830060 22724 tablet_bootstrap.cc:492] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:18.830246 22723 ts_tablet_manager.cc:1367] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.002s
I20240501 20:09:18.830317 22724 ts_tablet_manager.cc:1367] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20240501 20:09:18.831079 22723 raft_consensus.cc:348] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.831341 22723 raft_consensus.cc:374] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:18.831454 22723 raft_consensus.cc:729] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:18.831477 22726 tablet_bootstrap.cc:492] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:18.831682 22726 ts_tablet_manager.cc:1367] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.010s
I20240501 20:09:18.831578 22724 raft_consensus.cc:348] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.831779 22723 consensus_queue.cc:260] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.832237 22724 raft_consensus.cc:374] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:18.832399 22724 raft_consensus.cc:729] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:18.832490 22726 raft_consensus.cc:348] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:18.832768 22726 raft_consensus.cc:374] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:18.832906 22726 raft_consensus.cc:729] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:18.833182 22723 ts_tablet_manager.cc:1398] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:18.833217 22726 consensus_queue.cc:260] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:18.833508 22723 tablet_bootstrap.cc:492] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:18.833938 22726 ts_tablet_manager.cc:1398] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:18.832772 22724 consensus_queue.cc:260] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.836141 22723 tablet_bootstrap.cc:654] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:18.836426 22724 ts_tablet_manager.cc:1398] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:18.838303 22722 tablet_bootstrap.cc:492] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:18.838507 22722 ts_tablet_manager.cc:1367] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20240501 20:09:18.839362 22722 raft_consensus.cc:348] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:18.839601 22723 tablet_bootstrap.cc:492] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:18.839614 22722 raft_consensus.cc:374] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:18.839686 22722 raft_consensus.cc:729] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:18.839744 22723 ts_tablet_manager.cc:1367] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 20:09:18.839962 22722 consensus_queue.cc:260] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:18.840505 22722 ts_tablet_manager.cc:1398] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:18.840507 22723 raft_consensus.cc:348] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:18.840770 22723 raft_consensus.cc:374] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:18.840829 22722 tablet_bootstrap.cc:492] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:18.841136 22723 raft_consensus.cc:729] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:18.841456 22723 consensus_queue.cc:260] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:18.842092 22723 ts_tablet_manager.cc:1398] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:18.842623 22722 tablet_bootstrap.cc:654] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:18.844601 22722 tablet_bootstrap.cc:492] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:18.844738 22722 ts_tablet_manager.cc:1367] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:18.845302 22722 raft_consensus.cc:348] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.845705 22722 raft_consensus.cc:374] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:18.845825 22722 raft_consensus.cc:729] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:18.846072 22722 consensus_queue.cc:260] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.846534 22722 ts_tablet_manager.cc:1398] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:18.894346 22615 raft_consensus.cc:482] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:18.894516 22615 raft_consensus.cc:504] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.895277 22615 leader_election.cc:288] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:18.895814 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e80df9af0e40c5acc4dea95d7e95a8" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:18.896104 21356 raft_consensus.cc:2454] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:18.896066 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e80df9af0e40c5acc4dea95d7e95a8" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:18.896343 21097 raft_consensus.cc:2454] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:18.896627 21162 leader_election.cc:302] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:18.896899 22615 raft_consensus.cc:2781] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:18.897023 22615 raft_consensus.cc:482] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:18.897101 22615 raft_consensus.cc:3037] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:18.899914 22615 raft_consensus.cc:504] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.900655 22615 leader_election.cc:288] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:18.901005 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e80df9af0e40c5acc4dea95d7e95a8" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:18.901079 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e80df9af0e40c5acc4dea95d7e95a8" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:18.901213 21356 raft_consensus.cc:3037] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:18.901309 21097 raft_consensus.cc:3037] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:18.903046 21356 raft_consensus.cc:2454] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:18.903496 21162 leader_election.cc:302] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:18.903798 22615 raft_consensus.cc:2781] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:18.903956 22615 raft_consensus.cc:686] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:18.904044 21097 raft_consensus.cc:2454] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:18.904255 22615 consensus_queue.cc:237] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:18.907328 20821 catalog_manager.cc:5484] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:18.946748 22704 raft_consensus.cc:482] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:18.946978 22704 raft_consensus.cc:504] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:18.947729 22704 leader_election.cc:288] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:18.948239 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa02aed57a7a4e6a95b149d2078dc265" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:18.948331 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa02aed57a7a4e6a95b149d2078dc265" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:18.948484 21097 raft_consensus.cc:2454] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:18.948556 21356 raft_consensus.cc:2454] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:18.948949 20901 leader_election.cc:302] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:18.949235 22704 raft_consensus.cc:2781] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:18.949371 22704 raft_consensus.cc:482] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:18.949455 22704 raft_consensus.cc:3037] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:18.951445 22704 raft_consensus.cc:504] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:18.952100 22704 leader_election.cc:288] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:18.952589 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa02aed57a7a4e6a95b149d2078dc265" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:18.952718 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa02aed57a7a4e6a95b149d2078dc265" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:18.952796 21097 raft_consensus.cc:3037] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:18.952977 21356 raft_consensus.cc:3037] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:18.954716 21097 raft_consensus.cc:2454] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:18.954838 21356 raft_consensus.cc:2454] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:18.955199 20900 leader_election.cc:302] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:18.955509 22704 raft_consensus.cc:2781] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:18.955682 22704 raft_consensus.cc:686] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:18.955936 22704 consensus_queue.cc:237] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:18.959219 20821 catalog_manager.cc:5484] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:19.101421 22654 raft_consensus.cc:482] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:19.101608 22654 raft_consensus.cc:504] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:19.102248 22654 leader_election.cc:288] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:19.102684 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a115fe66d11143ea8e24aa891e4e2f33" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:19.102761 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a115fe66d11143ea8e24aa891e4e2f33" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:19.102948 21356 raft_consensus.cc:2454] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:19.103011 21226 raft_consensus.cc:2454] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:19.103344 21032 leader_election.cc:302] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:19.103664 22654 raft_consensus.cc:2781] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:19.103760 22654 raft_consensus.cc:482] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:19.103843 22654 raft_consensus.cc:3037] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:19.105836 22654 raft_consensus.cc:504] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:19.106474 22654 leader_election.cc:288] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:19.106904 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a115fe66d11143ea8e24aa891e4e2f33" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:19.106948 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a115fe66d11143ea8e24aa891e4e2f33" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:19.107112 21356 raft_consensus.cc:3037] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:19.107123 21226 raft_consensus.cc:3037] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:19.109401 21226 raft_consensus.cc:2454] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:19.109714 21356 raft_consensus.cc:2454] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:19.109777 21034 leader_election.cc:302] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:19.110044 22654 raft_consensus.cc:2781] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:19.110217 22654 raft_consensus.cc:686] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:19.110486 22654 consensus_queue.cc:237] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:19.113588 20821 catalog_manager.cc:5484] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:19.354589 22615 consensus_queue.cc:1035] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:19.368012 22615 consensus_queue.cc:1035] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:19.450680 22704 consensus_queue.cc:1035] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:19.464071 22704 consensus_queue.cc:1035] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:19.528128 22738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:19.528548 22738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:19.536285 22738 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:19.547230 22654 consensus_queue.cc:1035] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:19.552892 22654 consensus_queue.cc:1035] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 263.502 ms
time per row: 0.131751 ms
I20240501 20:09:19.953226 20819 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:19.961514 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb71fbd8e6d4de3948894bf939aaa65 (DEFAULT_TABLE table=rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.962016 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb71fbd8e6d4de3948894bf939aaa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.962327 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 71a423af43c04e659a0fa268afe10250 (DEFAULT_TABLE table=rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.962754 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a423af43c04e659a0fa268afe10250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.963014 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb71fbd8e6d4de3948894bf939aaa65 (DEFAULT_TABLE table=rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.963397 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb71fbd8e6d4de3948894bf939aaa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.967958 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 45083ec2f79e41559827ebce282688c9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.967972 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb71fbd8e6d4de3948894bf939aaa65 (DEFAULT_TABLE table=rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.968309 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45083ec2f79e41559827ebce282688c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.968416 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb71fbd8e6d4de3948894bf939aaa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.968997 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 71a423af43c04e659a0fa268afe10250 (DEFAULT_TABLE table=rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.969406 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a423af43c04e659a0fa268afe10250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.972064 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 45083ec2f79e41559827ebce282688c9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.972600 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45083ec2f79e41559827ebce282688c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.976639 22807 tablet_bootstrap.cc:492] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:19.978858 22807 tablet_bootstrap.cc:654] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:19.980644 22808 tablet_bootstrap.cc:492] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:19.982208 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 71a423af43c04e659a0fa268afe10250 (DEFAULT_TABLE table=rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.982676 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a423af43c04e659a0fa268afe10250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.983203 22808 tablet_bootstrap.cc:654] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:19.986193 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 45083ec2f79e41559827ebce282688c9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:19.986573 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45083ec2f79e41559827ebce282688c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:19.991837 22808 tablet_bootstrap.cc:492] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:19.992053 22808 ts_tablet_manager.cc:1367] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240501 20:09:19.992890 22808 raft_consensus.cc:348] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:19.993134 22808 raft_consensus.cc:374] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:19.993237 22808 raft_consensus.cc:729] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:19.993501 22808 consensus_queue.cc:260] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:19.994307 22808 ts_tablet_manager.cc:1398] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:19.994606 22808 tablet_bootstrap.cc:492] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:19.996383 22808 tablet_bootstrap.cc:654] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:20.003460 22810 tablet_bootstrap.cc:492] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:20.005302 22810 tablet_bootstrap.cc:654] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:20.007012 22808 tablet_bootstrap.cc:492] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:20.007251 22808 ts_tablet_manager.cc:1367] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240501 20:09:20.008134 22808 raft_consensus.cc:348] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:20.008428 22808 raft_consensus.cc:374] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:20.008561 22808 raft_consensus.cc:729] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:20.008890 22808 consensus_queue.cc:260] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:20.009603 22808 ts_tablet_manager.cc:1398] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:20.010100 22807 tablet_bootstrap.cc:492] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:20.010294 22807 ts_tablet_manager.cc:1367] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.004s
I20240501 20:09:20.010489 22808 tablet_bootstrap.cc:492] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:20.011093 22807 raft_consensus.cc:348] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:20.011320 22807 raft_consensus.cc:374] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:20.011425 22807 raft_consensus.cc:729] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:20.011719 22807 consensus_queue.cc:260] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:20.012373 22807 ts_tablet_manager.cc:1398] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:20.012701 22807 tablet_bootstrap.cc:492] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:20.014235 22807 tablet_bootstrap.cc:654] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:20.016803 22808 tablet_bootstrap.cc:654] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:20.022058 22814 tablet_bootstrap.cc:492] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:20.024003 22814 tablet_bootstrap.cc:654] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:20.024776 22807 tablet_bootstrap.cc:492] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:20.025005 22807 ts_tablet_manager.cc:1367] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20240501 20:09:20.025802 22807 raft_consensus.cc:348] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:20.026026 22807 raft_consensus.cc:374] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:20.026096 22807 raft_consensus.cc:729] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:20.026358 22814 tablet_bootstrap.cc:492] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:20.026346 22807 consensus_queue.cc:260] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:20.026500 22814 ts_tablet_manager.cc:1367] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:20.027020 22807 ts_tablet_manager.cc:1398] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:20.027242 22814 raft_consensus.cc:348] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:20.027390 22807 tablet_bootstrap.cc:492] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:20.027524 22814 raft_consensus.cc:374] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:20.027752 22814 raft_consensus.cc:729] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:20.028077 22814 consensus_queue.cc:260] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:20.028856 22807 tablet_bootstrap.cc:654] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:20.029678 22814 ts_tablet_manager.cc:1398] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:20.032734 22808 tablet_bootstrap.cc:492] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:20.032958 22808 ts_tablet_manager.cc:1367] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20240501 20:09:20.033993 22808 raft_consensus.cc:348] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:20.034279 22808 raft_consensus.cc:374] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:20.034380 22808 raft_consensus.cc:729] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:20.034644 22808 consensus_queue.cc:260] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:20.035372 22808 ts_tablet_manager.cc:1398] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:20.036737 22807 tablet_bootstrap.cc:492] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:20.036934 22807 ts_tablet_manager.cc:1367] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240501 20:09:20.037026 22810 tablet_bootstrap.cc:492] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:20.037209 22810 ts_tablet_manager.cc:1367] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.005s
I20240501 20:09:20.037703 22807 raft_consensus.cc:348] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:20.037887 22807 raft_consensus.cc:374] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:20.037966 22807 raft_consensus.cc:729] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:20.038023 22810 raft_consensus.cc:348] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:20.038211 22810 raft_consensus.cc:374] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:20.038307 22810 raft_consensus.cc:729] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:20.038223 22807 consensus_queue.cc:260] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:20.038512 22810 consensus_queue.cc:260] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:20.038949 22807 ts_tablet_manager.cc:1398] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:20.039144 22810 ts_tablet_manager.cc:1398] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:20.039425 22810 tablet_bootstrap.cc:492] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:20.042675 22810 tablet_bootstrap.cc:654] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:20.047843 22810 tablet_bootstrap.cc:492] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:20.048055 22810 ts_tablet_manager.cc:1367] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:09:20.048743 22810 raft_consensus.cc:348] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:20.048925 22810 raft_consensus.cc:374] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:20.048995 22810 raft_consensus.cc:729] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:20.049252 22810 consensus_queue.cc:260] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:20.049674 22810 ts_tablet_manager.cc:1398] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:20.135066 22654 raft_consensus.cc:482] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:20.135282 22654 raft_consensus.cc:504] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:20.136005 22654 leader_election.cc:288] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:20.136584 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb71fbd8e6d4de3948894bf939aaa65" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:20.136631 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb71fbd8e6d4de3948894bf939aaa65" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:20.136831 21226 raft_consensus.cc:2454] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:20.136883 20965 raft_consensus.cc:2454] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:20.137305 21031 leader_election.cc:302] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:20.137589 22654 raft_consensus.cc:2781] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:20.137712 22654 raft_consensus.cc:482] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:20.137801 22654 raft_consensus.cc:3037] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.139803 22654 raft_consensus.cc:504] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:20.140373 22654 leader_election.cc:288] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:20.140790 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb71fbd8e6d4de3948894bf939aaa65" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:20.140877 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb71fbd8e6d4de3948894bf939aaa65" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:20.141001 21226 raft_consensus.cc:3037] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.141079 20965 raft_consensus.cc:3037] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.142436 22651 raft_consensus.cc:482] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:20.142606 22651 raft_consensus.cc:504] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:20.143690 22651 leader_election.cc:288] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:20.143932 20965 raft_consensus.cc:2454] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:20.143960 21226 raft_consensus.cc:2454] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:20.144059 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a423af43c04e659a0fa268afe10250" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:20.144299 21228 raft_consensus.cc:2454] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:20.144393 21031 leader_election.cc:302] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:20.144668 22654 raft_consensus.cc:2781] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:20.144838 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a423af43c04e659a0fa268afe10250" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:20.145088 21358 raft_consensus.cc:2454] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:20.145161 20903 leader_election.cc:302] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:20.145381 22651 raft_consensus.cc:2781] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:20.145535 22651 raft_consensus.cc:482] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:20.145658 22651 raft_consensus.cc:3037] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.145601 22654 raft_consensus.cc:686] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:20.146047 22654 consensus_queue.cc:237] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:20.147480 22651 raft_consensus.cc:504] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:20.148093 22651 leader_election.cc:288] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:20.148600 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a423af43c04e659a0fa268afe10250" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:20.148800 21358 raft_consensus.cc:3037] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.148859 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a423af43c04e659a0fa268afe10250" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:20.149055 21228 raft_consensus.cc:3037] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.149586 20824 catalog_manager.cc:5484] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:20.150758 21358 raft_consensus.cc:2454] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:20.151131 20901 leader_election.cc:302] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:20.151656 21228 raft_consensus.cc:2454] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:20.151705 22651 raft_consensus.cc:2781] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:20.151825 22651 raft_consensus.cc:686] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:20.152087 22651 consensus_queue.cc:237] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:20.154945 20824 catalog_manager.cc:5484] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:20.326445 22796 raft_consensus.cc:482] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:20.326653 22796 raft_consensus.cc:504] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:20.327347 22796 leader_election.cc:288] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:20.327858 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45083ec2f79e41559827ebce282688c9" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:20.327934 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45083ec2f79e41559827ebce282688c9" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:20.328061 21228 raft_consensus.cc:2454] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:20.328182 21098 raft_consensus.cc:2454] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:20.328476 20903 leader_election.cc:302] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:20.328712 22796 raft_consensus.cc:2781] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:20.328809 22796 raft_consensus.cc:482] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:20.328872 22796 raft_consensus.cc:3037] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.330701 22796 raft_consensus.cc:504] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:20.331264 22796 leader_election.cc:288] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:20.331724 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45083ec2f79e41559827ebce282688c9" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:20.331743 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45083ec2f79e41559827ebce282688c9" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:20.331962 21098 raft_consensus.cc:3037] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.331964 21228 raft_consensus.cc:3037] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.334822 21228 raft_consensus.cc:2454] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:20.334838 21098 raft_consensus.cc:2454] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:20.335305 20900 leader_election.cc:302] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:20.335606 22796 raft_consensus.cc:2781] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:20.335754 22796 raft_consensus.cc:686] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:20.336036 22796 consensus_queue.cc:237] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:20.339146 20824 catalog_manager.cc:5484] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:20.657968 22651 consensus_queue.cc:1035] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:20.667793 22796 consensus_queue.cc:1035] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:20.718183 22824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:20.718529 22824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:09:20.720749 22654 consensus_queue.cc:1035] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:20.728245 22824 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:20.732952 22654 consensus_queue.cc:1035] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
I20240501 20:09:20.889286 21098 raft_consensus.cc:1261] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:20.890010 22796 consensus_queue.cc:1035] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:20.894869 21226 raft_consensus.cc:1261] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:20.895422 22796 consensus_queue.cc:1035] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.219 ms
time per row: 0.115109 ms
I20240501 20:09:21.143414 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:21.153360 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 75fa04c8726c4f2e932740cc6fbdcdd0 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:21.153378 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 75fa04c8726c4f2e932740cc6fbdcdd0 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:21.153831 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75fa04c8726c4f2e932740cc6fbdcdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.153831 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75fa04c8726c4f2e932740cc6fbdcdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.154515 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 4587a9d5392742b58ec8d64619f2dda1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:21.154884 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4587a9d5392742b58ec8d64619f2dda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.155974 21076 tablet_service.cc:1467] Processing CreateTablet for tablet ed9b47caf0a041fd962e4f18b0de5ff2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:21.156379 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9b47caf0a041fd962e4f18b0de5ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.158071 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 75fa04c8726c4f2e932740cc6fbdcdd0 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:21.158488 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75fa04c8726c4f2e932740cc6fbdcdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.164062 21338 tablet_service.cc:1467] Processing CreateTablet for tablet ed9b47caf0a041fd962e4f18b0de5ff2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:21.164574 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9b47caf0a041fd962e4f18b0de5ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.169013 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 4587a9d5392742b58ec8d64619f2dda1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:21.169482 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4587a9d5392742b58ec8d64619f2dda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.169816 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 4587a9d5392742b58ec8d64619f2dda1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:21.170230 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4587a9d5392742b58ec8d64619f2dda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.170711 22886 tablet_bootstrap.cc:492] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:21.172564 22886 tablet_bootstrap.cc:654] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.175796 21207 tablet_service.cc:1467] Processing CreateTablet for tablet ed9b47caf0a041fd962e4f18b0de5ff2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:21.176187 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9b47caf0a041fd962e4f18b0de5ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.177160 22890 tablet_bootstrap.cc:492] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:21.178959 22890 tablet_bootstrap.cc:654] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.180807 22887 tablet_bootstrap.cc:492] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:21.182483 22886 tablet_bootstrap.cc:492] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:21.182639 22886 ts_tablet_manager.cc:1367] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240501 20:09:21.183161 22887 tablet_bootstrap.cc:654] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.183475 22886 raft_consensus.cc:348] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:21.183830 22886 raft_consensus.cc:374] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.184000 22886 raft_consensus.cc:729] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.184321 22886 consensus_queue.cc:260] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:21.186512 22886 ts_tablet_manager.cc:1398] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 20:09:21.186931 22886 tablet_bootstrap.cc:492] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:21.187930 22888 tablet_bootstrap.cc:492] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:21.189710 22888 tablet_bootstrap.cc:654] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.190709 22886 tablet_bootstrap.cc:654] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.193159 22890 tablet_bootstrap.cc:492] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:21.193320 22890 ts_tablet_manager.cc:1367] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240501 20:09:21.193542 22887 tablet_bootstrap.cc:492] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:21.193745 22887 ts_tablet_manager.cc:1367] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240501 20:09:21.194025 22890 raft_consensus.cc:348] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:21.194196 22890 raft_consensus.cc:374] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.194303 22890 raft_consensus.cc:729] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.194540 22890 consensus_queue.cc:260] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:21.194761 22887 raft_consensus.cc:348] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:21.195070 22887 raft_consensus.cc:374] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.195204 22890 ts_tablet_manager.cc:1398] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:21.195217 22887 raft_consensus.cc:729] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.195511 22890 tablet_bootstrap.cc:492] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:21.195520 22887 consensus_queue.cc:260] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:21.196311 22887 ts_tablet_manager.cc:1398] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:09:21.196727 22887 tablet_bootstrap.cc:492] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:21.199667 22887 tablet_bootstrap.cc:654] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.199968 22890 tablet_bootstrap.cc:654] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.202931 22887 tablet_bootstrap.cc:492] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:21.202966 22888 tablet_bootstrap.cc:492] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:21.203114 22888 ts_tablet_manager.cc:1367] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20240501 20:09:21.203133 22887 ts_tablet_manager.cc:1367] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20240501 20:09:21.204038 22888 raft_consensus.cc:348] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:21.204017 22887 raft_consensus.cc:348] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:21.204233 22888 raft_consensus.cc:374] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.204259 22887 raft_consensus.cc:374] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.204325 22888 raft_consensus.cc:729] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.204331 22887 raft_consensus.cc:729] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.204542 22888 consensus_queue.cc:260] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:21.204591 22887 consensus_queue.cc:260] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:21.205209 22887 ts_tablet_manager.cc:1398] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:21.205531 22887 tablet_bootstrap.cc:492] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:21.206588 22888 ts_tablet_manager.cc:1398] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:21.206872 22888 tablet_bootstrap.cc:492] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:21.207237 22887 tablet_bootstrap.cc:654] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.208150 22886 tablet_bootstrap.cc:492] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:21.208389 22886 ts_tablet_manager.cc:1367] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240501 20:09:21.209460 22886 raft_consensus.cc:348] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:21.209772 22886 raft_consensus.cc:374] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.209864 22886 raft_consensus.cc:729] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.210105 22886 consensus_queue.cc:260] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:21.210620 22886 ts_tablet_manager.cc:1398] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:21.216951 22888 tablet_bootstrap.cc:654] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.217344 22887 tablet_bootstrap.cc:492] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:21.217536 22887 ts_tablet_manager.cc:1367] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240501 20:09:21.217914 22890 tablet_bootstrap.cc:492] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:21.218165 22890 ts_tablet_manager.cc:1367] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.005s
I20240501 20:09:21.218349 22887 raft_consensus.cc:348] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:21.218647 22887 raft_consensus.cc:374] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.218780 22887 raft_consensus.cc:729] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.218829 22890 raft_consensus.cc:348] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:21.218999 22888 tablet_bootstrap.cc:492] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:21.219085 22890 raft_consensus.cc:374] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.219168 22890 raft_consensus.cc:729] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.219184 22888 ts_tablet_manager.cc:1367] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 20:09:21.219095 22887 consensus_queue.cc:260] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:21.219414 22890 consensus_queue.cc:260] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:21.219835 22887 ts_tablet_manager.cc:1398] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:21.220019 22888 raft_consensus.cc:348] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:21.220191 22888 raft_consensus.cc:374] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.220235 22888 raft_consensus.cc:729] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.220448 22888 consensus_queue.cc:260] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:21.221105 22890 ts_tablet_manager.cc:1398] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:21.223235 22888 ts_tablet_manager.cc:1398] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:21.237358 22709 raft_consensus.cc:482] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:21.237542 22709 raft_consensus.cc:504] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:21.238435 22709 leader_election.cc:288] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:21.238902 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4587a9d5392742b58ec8d64619f2dda1" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:21.239053 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4587a9d5392742b58ec8d64619f2dda1" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:21.239163 21228 raft_consensus.cc:2454] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:21.239284 20965 raft_consensus.cc:2454] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:21.239755 21294 leader_election.cc:302] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:21.240031 22709 raft_consensus.cc:2781] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:21.240170 22709 raft_consensus.cc:482] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:21.240255 22709 raft_consensus.cc:3037] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.242182 22709 raft_consensus.cc:504] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:21.242785 22709 leader_election.cc:288] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:21.243285 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4587a9d5392742b58ec8d64619f2dda1" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:21.243319 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4587a9d5392742b58ec8d64619f2dda1" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:21.243513 20965 raft_consensus.cc:3037] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.243513 21228 raft_consensus.cc:3037] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.246376 20965 raft_consensus.cc:2454] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:21.246374 21228 raft_consensus.cc:2454] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:21.246767 21291 leader_election.cc:302] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:21.247058 22709 raft_consensus.cc:2781] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:21.247519 22709 raft_consensus.cc:686] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:21.247934 22709 consensus_queue.cc:237] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:21.251025 20826 catalog_manager.cc:5484] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:21.304469 22872 raft_consensus.cc:482] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:21.304713 22872 raft_consensus.cc:504] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:21.305531 22872 leader_election.cc:288] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:21.305997 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75fa04c8726c4f2e932740cc6fbdcdd0" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:21.306119 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75fa04c8726c4f2e932740cc6fbdcdd0" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:21.306237 21356 raft_consensus.cc:2454] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:21.306377 21097 raft_consensus.cc:2454] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:21.306738 20901 leader_election.cc:302] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:21.307029 22872 raft_consensus.cc:2781] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:21.307143 22872 raft_consensus.cc:482] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:21.307222 22872 raft_consensus.cc:3037] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.309221 22872 raft_consensus.cc:504] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:21.309942 22872 leader_election.cc:288] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:21.310304 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75fa04c8726c4f2e932740cc6fbdcdd0" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:21.310525 21356 raft_consensus.cc:3037] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.310530 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75fa04c8726c4f2e932740cc6fbdcdd0" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:21.310761 21097 raft_consensus.cc:3037] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.312510 21356 raft_consensus.cc:2454] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:21.312826 21097 raft_consensus.cc:2454] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:21.312992 20901 leader_election.cc:302] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:21.313361 22872 raft_consensus.cc:2781] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:21.313530 22872 raft_consensus.cc:686] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:21.313822 22872 consensus_queue.cc:237] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:21.317402 20826 catalog_manager.cc:5484] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:21.323105 22709 raft_consensus.cc:482] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:21.323307 22709 raft_consensus.cc:504] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:21.324136 22709 leader_election.cc:288] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:21.324590 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9b47caf0a041fd962e4f18b0de5ff2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:21.324743 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9b47caf0a041fd962e4f18b0de5ff2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:21.324824 21097 raft_consensus.cc:2454] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:21.324981 21228 raft_consensus.cc:2454] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:21.325295 21291 leader_election.cc:302] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:21.325623 22709 raft_consensus.cc:2781] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:21.325721 22709 raft_consensus.cc:482] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:21.325853 22709 raft_consensus.cc:3037] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.328115 22709 raft_consensus.cc:504] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:21.328660 22709 leader_election.cc:288] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:21.329092 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9b47caf0a041fd962e4f18b0de5ff2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:21.329106 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9b47caf0a041fd962e4f18b0de5ff2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:21.329299 21097 raft_consensus.cc:3037] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.329324 21228 raft_consensus.cc:3037] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.332259 21228 raft_consensus.cc:2454] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:21.332278 21097 raft_consensus.cc:2454] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:21.332715 21291 leader_election.cc:302] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:21.332986 22709 raft_consensus.cc:2781] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:21.333155 22709 raft_consensus.cc:686] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:21.333616 22709 consensus_queue.cc:237] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:21.336489 20826 catalog_manager.cc:5484] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:21.642963 22709 consensus_queue.cc:1035] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:21.648295 22709 consensus_queue.cc:1035] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:21.707383 22904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:21.707780 22904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:21.715955 22904 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 20:09:21.765765 22709 consensus_queue.cc:1035] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:21.779824 22872 consensus_queue.cc:1035] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:21.783707 22900 consensus_queue.cc:1035] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:21.800113 22869 consensus_queue.cc:1035] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 284.389 ms
time per row: 0.142194 ms
I20240501 20:09:22.155421 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:22.164285 21076 tablet_service.cc:1467] Processing CreateTablet for tablet f832bc442c3a416e83d55eab263ea0ea (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:22.164291 20947 tablet_service.cc:1467] Processing CreateTablet for tablet f832bc442c3a416e83d55eab263ea0ea (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:22.164744 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f832bc442c3a416e83d55eab263ea0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.164742 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f832bc442c3a416e83d55eab263ea0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.165555 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 473185b99beb4995a9089fc7c3a42ffe (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:22.165994 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473185b99beb4995a9089fc7c3a42ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.167999 21208 tablet_service.cc:1467] Processing CreateTablet for tablet bd4e9ca0289f44bf881f9e5078f76f4f (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:22.168397 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4e9ca0289f44bf881f9e5078f76f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.170840 21337 tablet_service.cc:1467] Processing CreateTablet for tablet f832bc442c3a416e83d55eab263ea0ea (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:22.171247 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f832bc442c3a416e83d55eab263ea0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.171981 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 473185b99beb4995a9089fc7c3a42ffe (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:22.174363 21077 tablet_service.cc:1467] Processing CreateTablet for tablet bd4e9ca0289f44bf881f9e5078f76f4f (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:22.174773 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4e9ca0289f44bf881f9e5078f76f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.175957 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473185b99beb4995a9089fc7c3a42ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.180730 20946 tablet_service.cc:1467] Processing CreateTablet for tablet bd4e9ca0289f44bf881f9e5078f76f4f (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:22.181354 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4e9ca0289f44bf881f9e5078f76f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.183631 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 473185b99beb4995a9089fc7c3a42ffe (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:22.184159 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473185b99beb4995a9089fc7c3a42ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:22.200520 22971 tablet_bootstrap.cc:492] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:22.200723 22972 tablet_bootstrap.cc:492] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:22.201601 22970 tablet_bootstrap.cc:492] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:22.201941 22973 tablet_bootstrap.cc:492] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:22.202610 22972 tablet_bootstrap.cc:654] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.202919 22971 tablet_bootstrap.cc:654] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.203840 22970 tablet_bootstrap.cc:654] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.205607 22973 tablet_bootstrap.cc:654] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.206007 22972 tablet_bootstrap.cc:492] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:22.206280 22972 ts_tablet_manager.cc:1367] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:09:22.207352 22972 raft_consensus.cc:348] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:22.207803 22972 raft_consensus.cc:374] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.207965 22972 raft_consensus.cc:729] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.208832 22972 consensus_queue.cc:260] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:22.210211 22972 ts_tablet_manager.cc:1398] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:22.210798 22972 tablet_bootstrap.cc:492] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:22.211802 22971 tablet_bootstrap.cc:492] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:22.212038 22971 ts_tablet_manager.cc:1367] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 20:09:22.212707 22972 tablet_bootstrap.cc:654] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.212954 22971 raft_consensus.cc:348] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:22.213243 22971 raft_consensus.cc:374] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.213470 22971 raft_consensus.cc:729] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.213799 22971 consensus_queue.cc:260] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:22.215915 22971 ts_tablet_manager.cc:1398] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:22.224491 22971 tablet_bootstrap.cc:492] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:22.226627 22971 tablet_bootstrap.cc:654] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.226634 22973 tablet_bootstrap.cc:492] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:22.226827 22973 ts_tablet_manager.cc:1367] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.017s
I20240501 20:09:22.227926 22973 raft_consensus.cc:348] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:22.228224 22973 raft_consensus.cc:374] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.228334 22973 raft_consensus.cc:729] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.228634 22973 consensus_queue.cc:260] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:22.229367 22973 ts_tablet_manager.cc:1398] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:22.229671 22973 tablet_bootstrap.cc:492] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:22.231773 22970 tablet_bootstrap.cc:492] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:22.232178 22970 ts_tablet_manager.cc:1367] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.003s
I20240501 20:09:22.232759 22972 tablet_bootstrap.cc:492] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:22.232945 22972 ts_tablet_manager.cc:1367] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.004s
I20240501 20:09:22.233239 22973 tablet_bootstrap.cc:654] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.233052 22970 raft_consensus.cc:348] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:22.233513 22970 raft_consensus.cc:374] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.233613 22970 raft_consensus.cc:729] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.233791 22972 raft_consensus.cc:348] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:22.233826 22970 consensus_queue.cc:260] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:22.234021 22972 raft_consensus.cc:374] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.234099 22972 raft_consensus.cc:729] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.234520 22970 ts_tablet_manager.cc:1398] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:22.236148 22970 tablet_bootstrap.cc:492] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:22.236433 22972 consensus_queue.cc:260] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:22.237525 22971 tablet_bootstrap.cc:492] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:22.237751 22971 ts_tablet_manager.cc:1367] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 20:09:22.238659 22972 ts_tablet_manager.cc:1398] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:22.239892 22971 raft_consensus.cc:348] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:22.240144 22971 raft_consensus.cc:374] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.240233 22971 raft_consensus.cc:729] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.240486 22971 consensus_queue.cc:260] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:22.241031 22971 ts_tablet_manager.cc:1398] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:22.241353 22971 tablet_bootstrap.cc:492] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:22.241385 22973 tablet_bootstrap.cc:492] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:22.241636 22973 ts_tablet_manager.cc:1367] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20240501 20:09:22.242458 22973 raft_consensus.cc:348] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:22.242748 22973 raft_consensus.cc:374] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.242868 22973 raft_consensus.cc:729] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.243081 22971 tablet_bootstrap.cc:654] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.243336 22973 consensus_queue.cc:260] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:22.243911 22970 tablet_bootstrap.cc:654] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:22.244230 22973 ts_tablet_manager.cc:1398] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:22.251262 22970 tablet_bootstrap.cc:492] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:22.251586 22970 ts_tablet_manager.cc:1367] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240501 20:09:22.251629 22971 tablet_bootstrap.cc:492] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:22.251796 22971 ts_tablet_manager.cc:1367] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240501 20:09:22.252604 22970 raft_consensus.cc:348] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:22.252839 22970 raft_consensus.cc:374] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.252691 22971 raft_consensus.cc:348] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:22.252916 22970 raft_consensus.cc:729] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.252945 22971 raft_consensus.cc:374] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:22.253032 22971 raft_consensus.cc:729] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:22.253161 22970 consensus_queue.cc:260] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:22.253243 22971 consensus_queue.cc:260] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:22.253847 22971 ts_tablet_manager.cc:1398] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:22.253852 22970 ts_tablet_manager.cc:1398] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:22.273119 22835 raft_consensus.cc:482] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:22.273296 22835 raft_consensus.cc:504] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:22.274329 22835 leader_election.cc:288] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:22.274729 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4e9ca0289f44bf881f9e5078f76f4f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:22.274968 21228 raft_consensus.cc:2454] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:22.275038 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4e9ca0289f44bf881f9e5078f76f4f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:22.275254 20967 raft_consensus.cc:2454] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:22.275501 21034 leader_election.cc:302] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:22.275858 22835 raft_consensus.cc:2781] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:22.275982 22835 raft_consensus.cc:482] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:22.276069 22835 raft_consensus.cc:3037] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.278018 22835 raft_consensus.cc:504] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:22.278515 22835 leader_election.cc:288] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:22.278934 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4e9ca0289f44bf881f9e5078f76f4f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:22.278997 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4e9ca0289f44bf881f9e5078f76f4f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:22.279117 21228 raft_consensus.cc:3037] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.279203 20967 raft_consensus.cc:3037] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.280974 21228 raft_consensus.cc:2454] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:22.281069 20967 raft_consensus.cc:2454] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:22.281306 22957 raft_consensus.cc:482] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:22.281381 21034 leader_election.cc:302] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:22.281461 22957 raft_consensus.cc:504] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:22.281651 22835 raft_consensus.cc:2781] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:22.281884 22835 raft_consensus.cc:686] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:22.282131 22957 leader_election.cc:288] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:22.282213 22835 consensus_queue.cc:237] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:22.282649 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f832bc442c3a416e83d55eab263ea0ea" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:22.282804 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f832bc442c3a416e83d55eab263ea0ea" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:22.282974 20967 raft_consensus.cc:2454] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:22.283026 21098 raft_consensus.cc:2454] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:22.283347 21291 leader_election.cc:302] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:22.283640 22957 raft_consensus.cc:2781] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:22.283757 22957 raft_consensus.cc:482] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:22.283864 22957 raft_consensus.cc:3037] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.285792 22957 raft_consensus.cc:504] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:22.286305 22957 leader_election.cc:288] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:22.286847 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f832bc442c3a416e83d55eab263ea0ea" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:22.286909 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f832bc442c3a416e83d55eab263ea0ea" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:22.287040 21098 raft_consensus.cc:3037] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.287178 20965 raft_consensus.cc:3037] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.289129 21098 raft_consensus.cc:2454] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:22.290813 20965 raft_consensus.cc:2454] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:22.291683 21291 leader_election.cc:302] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:22.292080 22957 raft_consensus.cc:2781] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:22.292423 22957 raft_consensus.cc:686] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:22.292748 22957 consensus_queue.cc:237] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:22.294757 20826 catalog_manager.cc:5484] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:22.297637 20822 catalog_manager.cc:5484] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:22.336079 22615 raft_consensus.cc:482] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:22.336295 22615 raft_consensus.cc:504] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:22.337029 22615 leader_election.cc:288] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:22.337615 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473185b99beb4995a9089fc7c3a42ffe" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:22.337615 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473185b99beb4995a9089fc7c3a42ffe" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:22.337855 21356 raft_consensus.cc:2454] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:22.337855 20965 raft_consensus.cc:2454] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:22.338315 21162 leader_election.cc:302] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:22.338641 22615 raft_consensus.cc:2781] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:22.338748 22615 raft_consensus.cc:482] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:22.338891 22615 raft_consensus.cc:3037] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.341087 22615 raft_consensus.cc:504] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:22.341714 22615 leader_election.cc:288] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:22.342252 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473185b99beb4995a9089fc7c3a42ffe" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:22.342254 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473185b99beb4995a9089fc7c3a42ffe" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:22.342444 20965 raft_consensus.cc:3037] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.342445 21356 raft_consensus.cc:3037] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:22.344658 20965 raft_consensus.cc:2454] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:22.344676 21356 raft_consensus.cc:2454] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:22.345058 21161 leader_election.cc:302] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:22.345441 22615 raft_consensus.cc:2781] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:22.345825 22615 raft_consensus.cc:686] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:22.346190 22615 consensus_queue.cc:237] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:22.349397 20822 catalog_manager.cc:5484] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:22.669759 22835 consensus_queue.cc:1035] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:22.675644 22835 consensus_queue.cc:1035] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:22.711989 22957 consensus_queue.cc:1035] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:22.718501 22709 consensus_queue.cc:1035] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:22.753365 22987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:22.753719 22987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:22.761729 22987 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240501 20:09:22.795635 22615 consensus_queue.cc:1035] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:22.803476 22615 consensus_queue.cc:1035] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 329.924 ms
time per row: 0.164962 ms
I20240501 20:09:23.246269 20824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:23.255934 21076 tablet_service.cc:1467] Processing CreateTablet for tablet b4c4664b17f04e2d8285b7851b27d1a1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.256170 21207 tablet_service.cc:1467] Processing CreateTablet for tablet b4c4664b17f04e2d8285b7851b27d1a1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.256292 21338 tablet_service.cc:1467] Processing CreateTablet for tablet b70e144824074927bb0f4eafcdeadcec (DEFAULT_TABLE table=rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.256429 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c4664b17f04e2d8285b7851b27d1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.256582 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c4664b17f04e2d8285b7851b27d1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.256716 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70e144824074927bb0f4eafcdeadcec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.261828 21337 tablet_service.cc:1467] Processing CreateTablet for tablet b4c4664b17f04e2d8285b7851b27d1a1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.262229 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c4664b17f04e2d8285b7851b27d1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.263505 23056 tablet_bootstrap.cc:492] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:23.263684 23057 tablet_bootstrap.cc:492] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:23.265404 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 998f71aeae984fd5b866cb866bd4e87a (DEFAULT_TABLE table=rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.265487 21208 tablet_service.cc:1467] Processing CreateTablet for tablet b70e144824074927bb0f4eafcdeadcec (DEFAULT_TABLE table=rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.265708 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 998f71aeae984fd5b866cb866bd4e87a (DEFAULT_TABLE table=rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.265894 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70e144824074927bb0f4eafcdeadcec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.265883 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998f71aeae984fd5b866cb866bd4e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.266049 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998f71aeae984fd5b866cb866bd4e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.269328 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 998f71aeae984fd5b866cb866bd4e87a (DEFAULT_TABLE table=rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.269728 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998f71aeae984fd5b866cb866bd4e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.270334 23055 tablet_bootstrap.cc:492] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:23.272588 23057 tablet_bootstrap.cc:654] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.272706 23055 tablet_bootstrap.cc:654] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.279619 23056 tablet_bootstrap.cc:654] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.283421 23055 tablet_bootstrap.cc:492] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:23.283682 23055 ts_tablet_manager.cc:1367] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240501 20:09:23.284600 23055 raft_consensus.cc:348] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.284868 23055 raft_consensus.cc:374] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.284992 23055 raft_consensus.cc:729] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.285769 23055 consensus_queue.cc:260] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.288156 23055 ts_tablet_manager.cc:1398] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:09:23.288834 23056 tablet_bootstrap.cc:492] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:23.289155 23056 ts_tablet_manager.cc:1367] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20240501 20:09:23.291915 23056 raft_consensus.cc:348] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.292546 23056 raft_consensus.cc:374] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.292824 23056 raft_consensus.cc:729] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.292752 21076 tablet_service.cc:1467] Processing CreateTablet for tablet b70e144824074927bb0f4eafcdeadcec (DEFAULT_TABLE table=rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.293238 23056 consensus_queue.cc:260] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.293799 23056 ts_tablet_manager.cc:1398] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:09:23.295114 23055 tablet_bootstrap.cc:492] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:23.296319 23056 tablet_bootstrap.cc:492] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:23.298266 23056 tablet_bootstrap.cc:654] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.300897 23055 tablet_bootstrap.cc:654] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.302038 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70e144824074927bb0f4eafcdeadcec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.303355 23057 tablet_bootstrap.cc:492] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:23.303601 23057 ts_tablet_manager.cc:1367] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.027s
I20240501 20:09:23.304494 23056 tablet_bootstrap.cc:492] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:23.304560 23057 raft_consensus.cc:348] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.304785 23056 ts_tablet_manager.cc:1367] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:09:23.304868 23057 raft_consensus.cc:374] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.305023 23057 raft_consensus.cc:729] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.305415 23057 consensus_queue.cc:260] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.306113 23057 ts_tablet_manager.cc:1398] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:23.306028 23056 raft_consensus.cc:348] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:23.306347 23056 raft_consensus.cc:374] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.306500 23056 raft_consensus.cc:729] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.306589 23057 tablet_bootstrap.cc:492] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:23.306890 23056 consensus_queue.cc:260] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:23.307780 23056 ts_tablet_manager.cc:1398] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:23.311241 23057 tablet_bootstrap.cc:654] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.312750 23055 tablet_bootstrap.cc:492] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:23.313072 23055 ts_tablet_manager.cc:1367] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20240501 20:09:23.314275 23055 raft_consensus.cc:348] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.314806 23055 raft_consensus.cc:374] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.314996 23055 raft_consensus.cc:729] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.315407 23055 consensus_queue.cc:260] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.316248 23056 tablet_bootstrap.cc:492] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:23.317582 23055 ts_tablet_manager.cc:1398] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:23.318393 23060 tablet_bootstrap.cc:492] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:23.321758 23056 tablet_bootstrap.cc:654] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.324165 23060 tablet_bootstrap.cc:654] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.327309 23057 tablet_bootstrap.cc:492] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:23.327524 23057 ts_tablet_manager.cc:1367] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20240501 20:09:23.327746 23060 tablet_bootstrap.cc:492] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:23.327862 23056 tablet_bootstrap.cc:492] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:23.327906 23060 ts_tablet_manager.cc:1367] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 20:09:23.328020 23056 ts_tablet_manager.cc:1367] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.006s
I20240501 20:09:23.328423 23057 raft_consensus.cc:348] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:23.328665 23057 raft_consensus.cc:374] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.328756 23057 raft_consensus.cc:729] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.328631 23056 raft_consensus.cc:348] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.328684 23060 raft_consensus.cc:348] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:23.328845 23056 raft_consensus.cc:374] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.328922 23060 raft_consensus.cc:374] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.328972 23056 raft_consensus.cc:729] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.329030 23060 raft_consensus.cc:729] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.329046 23057 consensus_queue.cc:260] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:23.329239 23056 consensus_queue.cc:260] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.329280 23060 consensus_queue.cc:260] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:23.329691 23057 ts_tablet_manager.cc:1398] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:23.329993 23057 tablet_bootstrap.cc:492] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:23.330610 23056 ts_tablet_manager.cc:1398] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:23.330796 23060 ts_tablet_manager.cc:1398] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:23.332367 23057 tablet_bootstrap.cc:654] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.336469 23057 tablet_bootstrap.cc:492] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:23.336625 23057 ts_tablet_manager.cc:1367] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:09:23.337383 23057 raft_consensus.cc:348] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.337606 23057 raft_consensus.cc:374] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.337703 23057 raft_consensus.cc:729] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.337926 23057 consensus_queue.cc:260] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.338512 23057 ts_tablet_manager.cc:1398] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:23.366403 23038 raft_consensus.cc:482] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:23.366640 23038 raft_consensus.cc:504] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:23.367408 23038 leader_election.cc:288] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:23.367946 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998f71aeae984fd5b866cb866bd4e87a" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:23.367950 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998f71aeae984fd5b866cb866bd4e87a" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:23.368180 21098 raft_consensus.cc:2454] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:23.368187 20967 raft_consensus.cc:2454] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:23.368639 21291 leader_election.cc:302] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:23.368927 23038 raft_consensus.cc:2781] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:23.369025 23038 raft_consensus.cc:482] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:23.369117 23038 raft_consensus.cc:3037] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:23.371387 23038 raft_consensus.cc:504] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:23.371949 23038 leader_election.cc:288] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:23.372578 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998f71aeae984fd5b866cb866bd4e87a" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:23.372570 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998f71aeae984fd5b866cb866bd4e87a" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:23.372830 21098 raft_consensus.cc:3037] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:23.372830 20967 raft_consensus.cc:3037] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:23.375748 21098 raft_consensus.cc:2454] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:23.375748 20967 raft_consensus.cc:2454] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:23.376339 21291 leader_election.cc:302] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:23.376658 23038 raft_consensus.cc:2781] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:23.376806 23038 raft_consensus.cc:686] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:23.377090 23038 consensus_queue.cc:237] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:23.381003 20823 catalog_manager.cc:5484] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:23.453177 23038 raft_consensus.cc:482] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:23.453414 23038 raft_consensus.cc:504] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.454079 23038 leader_election.cc:288] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:23.454774 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c4664b17f04e2d8285b7851b27d1a1" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:23.454581 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c4664b17f04e2d8285b7851b27d1a1" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:23.455018 21228 raft_consensus.cc:2454] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:23.455132 21098 raft_consensus.cc:2454] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:23.455494 21294 leader_election.cc:302] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:23.455799 23038 raft_consensus.cc:2781] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:23.456235 23038 raft_consensus.cc:482] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:23.456383 23038 raft_consensus.cc:3037] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:23.458792 23038 raft_consensus.cc:504] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.459566 23038 leader_election.cc:288] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:23.459918 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c4664b17f04e2d8285b7851b27d1a1" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:23.459950 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c4664b17f04e2d8285b7851b27d1a1" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:23.460120 21228 raft_consensus.cc:3037] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:23.460184 21098 raft_consensus.cc:3037] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:23.462577 21228 raft_consensus.cc:2454] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:23.462958 21294 leader_election.cc:302] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:23.463197 23038 raft_consensus.cc:2781] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:23.463399 21098 raft_consensus.cc:2454] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:23.463466 23038 raft_consensus.cc:686] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:23.463776 23038 consensus_queue.cc:237] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.467020 20823 catalog_manager.cc:5484] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:23.520229 23038 raft_consensus.cc:482] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:23.520426 23038 raft_consensus.cc:504] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.521070 23038 leader_election.cc:288] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:23.521657 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70e144824074927bb0f4eafcdeadcec" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:23.521685 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70e144824074927bb0f4eafcdeadcec" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:23.521920 21098 raft_consensus.cc:2454] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:23.521914 21228 raft_consensus.cc:2454] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:23.522364 21291 leader_election.cc:302] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:23.522647 23038 raft_consensus.cc:2781] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:23.522766 23038 raft_consensus.cc:482] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:23.522845 23038 raft_consensus.cc:3037] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:23.525656 23038 raft_consensus.cc:504] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.526463 23038 leader_election.cc:288] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:23.526655 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70e144824074927bb0f4eafcdeadcec" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:23.526712 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70e144824074927bb0f4eafcdeadcec" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:23.526898 21097 raft_consensus.cc:3037] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:23.526885 21228 raft_consensus.cc:3037] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:23.529678 21097 raft_consensus.cc:2454] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:23.529691 21228 raft_consensus.cc:2454] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:23.530084 21294 leader_election.cc:302] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:23.530575 23038 raft_consensus.cc:2781] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:23.530747 23038 raft_consensus.cc:686] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:23.531077 23038 consensus_queue.cc:237] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:23.534884 20823 catalog_manager.cc:5484] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:23.814335 23038 consensus_queue.cc:1035] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:23.821481 23038 consensus_queue.cc:1035] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:23.898556 23075 consensus_queue.cc:1035] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:23.911013 23038 consensus_queue.cc:1035] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:23.965946 23071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:23.966276 23071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:23.975711 23071 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240501 20:09:24.043710 23038 consensus_queue.cc:1035] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:24.056697 23075 consensus_queue.cc:1035] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:24.193905 21294 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 53858.2us
INSERT report
rows total: 2000
time total: 241.418 ms
time per row: 0.120709 ms
I20240501 20:09:24.374336 20824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:24.383746 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 07a4fb0b08244e47ab330bafe40991bd (DEFAULT_TABLE table=rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:24.384073 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 07a4fb0b08244e47ab330bafe40991bd (DEFAULT_TABLE table=rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:24.384238 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a4fb0b08244e47ab330bafe40991bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:24.384524 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a4fb0b08244e47ab330bafe40991bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:24.385665 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 26c37bbdd8d74983a4966478aec924fe (DEFAULT_TABLE table=rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:24.386124 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c37bbdd8d74983a4966478aec924fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:24.386816 21207 tablet_service.cc:1467] Processing CreateTablet for tablet b5a05a1a27f54527b4d9bfa05faf5434 (DEFAULT_TABLE table=rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:24.387228 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a05a1a27f54527b4d9bfa05faf5434. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:24.387920 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 07a4fb0b08244e47ab330bafe40991bd (DEFAULT_TABLE table=rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:24.388343 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a4fb0b08244e47ab330bafe40991bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:24.391232 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 26c37bbdd8d74983a4966478aec924fe (DEFAULT_TABLE table=rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:24.391772 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c37bbdd8d74983a4966478aec924fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:24.391891 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 26c37bbdd8d74983a4966478aec924fe (DEFAULT_TABLE table=rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:24.392261 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c37bbdd8d74983a4966478aec924fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:24.396004 20946 tablet_service.cc:1467] Processing CreateTablet for tablet b5a05a1a27f54527b4d9bfa05faf5434 (DEFAULT_TABLE table=rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:24.396466 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a05a1a27f54527b4d9bfa05faf5434. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:24.397192 21337 tablet_service.cc:1467] Processing CreateTablet for tablet b5a05a1a27f54527b4d9bfa05faf5434 (DEFAULT_TABLE table=rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:24.397616 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a05a1a27f54527b4d9bfa05faf5434. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:24.401926 23136 tablet_bootstrap.cc:492] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:24.403914 23136 tablet_bootstrap.cc:654] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:24.410629 23138 tablet_bootstrap.cc:492] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:24.414129 23137 tablet_bootstrap.cc:492] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:24.414755 23138 tablet_bootstrap.cc:654] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:24.417047 23137 tablet_bootstrap.cc:654] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:24.418959 23137 tablet_bootstrap.cc:492] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:24.419184 23137 ts_tablet_manager.cc:1367] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:24.419737 23142 tablet_bootstrap.cc:492] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:24.420083 23137 raft_consensus.cc:348] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.420404 23137 raft_consensus.cc:374] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:24.420750 23137 raft_consensus.cc:729] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:24.421309 23137 consensus_queue.cc:260] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.421999 23142 tablet_bootstrap.cc:654] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:24.422525 23137 ts_tablet_manager.cc:1398] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:24.423449 23137 tablet_bootstrap.cc:492] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:24.425937 23137 tablet_bootstrap.cc:654] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:24.431020 23142 tablet_bootstrap.cc:492] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:24.431262 23142 ts_tablet_manager.cc:1367] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 20:09:24.432482 23142 raft_consensus.cc:348] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:24.432825 23142 raft_consensus.cc:374] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:24.432986 23142 raft_consensus.cc:729] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:24.433349 23142 consensus_queue.cc:260] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:24.434154 23142 ts_tablet_manager.cc:1398] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:24.435954 23137 tablet_bootstrap.cc:492] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:24.436554 23137 ts_tablet_manager.cc:1367] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240501 20:09:24.437567 23137 raft_consensus.cc:348] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.438702 23137 raft_consensus.cc:374] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:24.438966 23137 raft_consensus.cc:729] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:24.439453 23137 consensus_queue.cc:260] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.440510 23142 tablet_bootstrap.cc:492] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:24.440569 23137 ts_tablet_manager.cc:1398] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:24.442612 23142 tablet_bootstrap.cc:654] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:24.451270 23136 tablet_bootstrap.cc:492] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:24.451529 23136 ts_tablet_manager.cc:1367] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.050s user 0.028s sys 0.020s
I20240501 20:09:24.451761 23142 tablet_bootstrap.cc:492] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:24.451939 23142 ts_tablet_manager.cc:1367] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240501 20:09:24.452452 23136 raft_consensus.cc:348] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:24.452700 23136 raft_consensus.cc:374] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:24.452649 23142 raft_consensus.cc:348] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.452831 23136 raft_consensus.cc:729] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:24.452878 23142 raft_consensus.cc:374] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:24.452998 23142 raft_consensus.cc:729] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:24.453123 23136 consensus_queue.cc:260] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:24.453234 23142 consensus_queue.cc:260] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.453776 23136 ts_tablet_manager.cc:1398] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:24.454025 23142 ts_tablet_manager.cc:1398] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:24.454258 23136 tablet_bootstrap.cc:492] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:24.454304 23142 tablet_bootstrap.cc:492] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:24.455978 23136 tablet_bootstrap.cc:654] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:24.459198 23142 tablet_bootstrap.cc:654] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:24.460665 23138 tablet_bootstrap.cc:492] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:24.460955 23138 ts_tablet_manager.cc:1367] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.005s
I20240501 20:09:24.462065 23138 raft_consensus.cc:348] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:24.462352 23138 raft_consensus.cc:374] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:24.462460 23138 raft_consensus.cc:729] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:24.462858 23138 consensus_queue.cc:260] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:24.463811 23138 ts_tablet_manager.cc:1398] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:24.464206 23136 tablet_bootstrap.cc:492] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:24.464363 23136 ts_tablet_manager.cc:1367] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:09:24.465294 23136 raft_consensus.cc:348] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.465540 23136 raft_consensus.cc:374] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:24.465642 23136 raft_consensus.cc:729] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:24.465962 23136 consensus_queue.cc:260] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.469448 23136 ts_tablet_manager.cc:1398] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240501 20:09:24.469879 23136 tablet_bootstrap.cc:492] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:24.472234 23136 tablet_bootstrap.cc:654] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:24.474807 23142 tablet_bootstrap.cc:492] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:24.474937 23142 ts_tablet_manager.cc:1367] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20240501 20:09:24.475687 23142 raft_consensus.cc:348] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.475901 23142 raft_consensus.cc:374] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:24.475996 23142 raft_consensus.cc:729] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:24.476222 23142 consensus_queue.cc:260] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.476449 23136 tablet_bootstrap.cc:492] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:24.476569 23136 ts_tablet_manager.cc:1367] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240501 20:09:24.476756 23142 ts_tablet_manager.cc:1398] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:24.477380 23136 raft_consensus.cc:348] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.477571 23136 raft_consensus.cc:374] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:24.477665 23136 raft_consensus.cc:729] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:24.477924 23136 consensus_queue.cc:260] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.478514 23136 ts_tablet_manager.cc:1398] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:24.593464 23108 raft_consensus.cc:482] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:24.593664 23108 raft_consensus.cc:504] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.593777 22835 raft_consensus.cc:482] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:24.593986 22835 raft_consensus.cc:504] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.594436 23108 leader_election.cc:288] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:24.594619 22835 leader_election.cc:288] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:24.594909 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c37bbdd8d74983a4966478aec924fe" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:24.595036 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c37bbdd8d74983a4966478aec924fe" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:24.595120 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a4fb0b08244e47ab330bafe40991bd" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:24.595202 21228 raft_consensus.cc:2454] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:24.595242 21097 raft_consensus.cc:2454] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:24.595341 20965 raft_consensus.cc:2454] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:24.595680 20903 leader_election.cc:302] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:24.595892 21031 leader_election.cc:302] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:24.595988 23108 raft_consensus.cc:2781] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:24.596132 23108 raft_consensus.cc:482] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:24.596180 22835 raft_consensus.cc:2781] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:24.596277 22835 raft_consensus.cc:482] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:24.596273 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a4fb0b08244e47ab330bafe40991bd" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:24.596381 22835 raft_consensus.cc:3037] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.596478 21228 raft_consensus.cc:2454] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:24.596252 23108 raft_consensus.cc:3037] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.599072 22835 raft_consensus.cc:504] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.599241 23108 raft_consensus.cc:504] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.599736 22835 leader_election.cc:288] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:24.599893 23108 leader_election.cc:288] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:24.600189 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a4fb0b08244e47ab330bafe40991bd" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:24.600334 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c37bbdd8d74983a4966478aec924fe" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:24.600384 20965 raft_consensus.cc:3037] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.600286 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a4fb0b08244e47ab330bafe40991bd" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:24.600489 21097 raft_consensus.cc:3037] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.600416 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c37bbdd8d74983a4966478aec924fe" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:24.600632 21228 raft_consensus.cc:3037] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.600687 21226 raft_consensus.cc:3037] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.603266 20965 raft_consensus.cc:2454] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:24.603298 21097 raft_consensus.cc:2454] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:24.603637 21226 raft_consensus.cc:2454] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:24.603840 21031 leader_election.cc:302] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:24.603811 20900 leader_election.cc:302] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:24.604132 22835 raft_consensus.cc:2781] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:24.604153 23108 raft_consensus.cc:2781] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:24.604316 22835 raft_consensus.cc:686] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:24.604470 21228 raft_consensus.cc:2454] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:24.604717 23108 raft_consensus.cc:686] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:24.604663 22835 consensus_queue.cc:237] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.605031 23108 consensus_queue.cc:237] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:24.608527 20824 catalog_manager.cc:5484] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:24.609014 20819 catalog_manager.cc:5484] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:24.614790 23148 raft_consensus.cc:482] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:24.614972 23148 raft_consensus.cc:504] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:24.615784 23148 leader_election.cc:288] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:24.616073 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a05a1a27f54527b4d9bfa05faf5434" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:24.616070 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a05a1a27f54527b4d9bfa05faf5434" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:24.616276 20965 raft_consensus.cc:2454] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:24.616274 21358 raft_consensus.cc:2454] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:24.616688 21161 leader_election.cc:302] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:24.616971 23148 raft_consensus.cc:2781] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:24.617081 23148 raft_consensus.cc:482] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:24.617169 23148 raft_consensus.cc:3037] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.619089 23148 raft_consensus.cc:504] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:24.619688 23148 leader_election.cc:288] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:24.619992 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a05a1a27f54527b4d9bfa05faf5434" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:24.620229 21358 raft_consensus.cc:3037] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.620167 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a05a1a27f54527b4d9bfa05faf5434" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:24.620362 20965 raft_consensus.cc:3037] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.623018 21358 raft_consensus.cc:2454] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:24.623096 20965 raft_consensus.cc:2454] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:24.623473 21161 leader_election.cc:302] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:24.623811 23148 raft_consensus.cc:2781] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:24.623946 23148 raft_consensus.cc:686] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:24.624181 23148 consensus_queue.cc:237] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:24.628382 20819 catalog_manager.cc:5484] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
W20240501 20:09:25.019631 23153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:25.019976 23153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:09:25.025012 23148 consensus_queue.cc:1035] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:25.025115 22835 consensus_queue.cc:1035] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:25.030637 23153 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:25.053764 23148 consensus_queue.cc:1035] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:25.054525 23164 consensus_queue.cc:1035] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
I20240501 20:09:25.132218 22988 consensus_queue.cc:1035] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:25.137288 23108 consensus_queue.cc:1035] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 267.289 ms
time per row: 0.133644 ms
I20240501 20:09:25.470098 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:25.479336 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 4d14c3c71012405e81dabba5f127433d (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.479579 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 4d14c3c71012405e81dabba5f127433d (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.479890 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d14c3c71012405e81dabba5f127433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.480000 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d14c3c71012405e81dabba5f127433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.480094 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 30d2aada073b4d8989d7db8be4034756 (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.480445 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d2aada073b4d8989d7db8be4034756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.484390 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 30d2aada073b4d8989d7db8be4034756 (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.484462 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 30d2aada073b4d8989d7db8be4034756 (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.484444 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 4d14c3c71012405e81dabba5f127433d (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.484822 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d2aada073b4d8989d7db8be4034756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.484843 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d2aada073b4d8989d7db8be4034756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.485085 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d14c3c71012405e81dabba5f127433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.492648 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 748b4fd500f84a359bd8ee05b861300d (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.493144 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748b4fd500f84a359bd8ee05b861300d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.493469 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 748b4fd500f84a359bd8ee05b861300d (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.493891 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748b4fd500f84a359bd8ee05b861300d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.500892 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 748b4fd500f84a359bd8ee05b861300d (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:25.501363 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748b4fd500f84a359bd8ee05b861300d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:25.513567 23222 tablet_bootstrap.cc:492] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:25.516642 23222 tablet_bootstrap.cc:654] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.523885 23220 tablet_bootstrap.cc:492] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:25.526834 23220 tablet_bootstrap.cc:654] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.529996 23221 tablet_bootstrap.cc:492] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:25.532028 23220 tablet_bootstrap.cc:492] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:25.532241 23220 ts_tablet_manager.cc:1367] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:09:25.533134 23220 raft_consensus.cc:348] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.533471 23220 raft_consensus.cc:374] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.533519 23222 tablet_bootstrap.cc:492] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:25.533680 23220 raft_consensus.cc:729] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.533763 23222 ts_tablet_manager.cc:1367] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20240501 20:09:25.534096 23220 consensus_queue.cc:260] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.534530 23222 raft_consensus.cc:348] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.534647 23221 tablet_bootstrap.cc:654] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.534694 23222 raft_consensus.cc:374] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.534787 23222 raft_consensus.cc:729] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.535044 23222 consensus_queue.cc:260] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.535984 23220 ts_tablet_manager.cc:1398] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:25.536263 23220 tablet_bootstrap.cc:492] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:25.535514 23222 ts_tablet_manager.cc:1398] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:25.537015 23222 tablet_bootstrap.cc:492] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:25.538399 23222 tablet_bootstrap.cc:654] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.539430 23220 tablet_bootstrap.cc:654] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.544914 23223 tablet_bootstrap.cc:492] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:25.545336 22835 raft_consensus.cc:482] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:25.545521 22835 raft_consensus.cc:504] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.546356 22835 leader_election.cc:288] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:25.547082 23223 tablet_bootstrap.cc:654] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.547142 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d14c3c71012405e81dabba5f127433d" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:25.547178 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d14c3c71012405e81dabba5f127433d" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
W20240501 20:09:25.548166 21031 leader_election.cc:341] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:25.548527 21032 leader_election.cc:341] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:25.548655 21032 leader_election.cc:302] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feac73db0a3b4aafa374ca5dfd322b08; no voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0
I20240501 20:09:25.548974 22835 raft_consensus.cc:2726] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:25.550961 23222 tablet_bootstrap.cc:492] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:25.551136 23220 tablet_bootstrap.cc:492] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:25.551146 23222 ts_tablet_manager.cc:1367] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20240501 20:09:25.551276 23220 ts_tablet_manager.cc:1367] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20240501 20:09:25.551450 23223 tablet_bootstrap.cc:492] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:25.551622 23223 ts_tablet_manager.cc:1367] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240501 20:09:25.552031 23222 raft_consensus.cc:348] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.552156 23220 raft_consensus.cc:348] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:25.552253 23222 raft_consensus.cc:374] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.552346 23220 raft_consensus.cc:374] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.552338 23222 raft_consensus.cc:729] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.552402 23220 raft_consensus.cc:729] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.552403 23223 raft_consensus.cc:348] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:25.552598 23223 raft_consensus.cc:374] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.552682 23223 raft_consensus.cc:729] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.552598 23222 consensus_queue.cc:260] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.552676 23220 consensus_queue.cc:260] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:25.553139 23222 ts_tablet_manager.cc:1398] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:25.552897 23223 consensus_queue.cc:260] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:25.554352 23220 ts_tablet_manager.cc:1398] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:25.555227 23221 tablet_bootstrap.cc:492] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:25.555436 23221 ts_tablet_manager.cc:1367] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240501 20:09:25.556066 23223 ts_tablet_manager.cc:1398] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:25.556399 23223 tablet_bootstrap.cc:492] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:25.556318 23221 raft_consensus.cc:348] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:25.557601 23221 raft_consensus.cc:374] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.557754 23221 raft_consensus.cc:729] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.558192 23223 tablet_bootstrap.cc:654] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.558123 23221 consensus_queue.cc:260] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:25.558698 23221 ts_tablet_manager.cc:1398] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:25.560775 23221 tablet_bootstrap.cc:492] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:25.562496 23221 tablet_bootstrap.cc:654] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.575042 23221 tablet_bootstrap.cc:492] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:25.575234 23221 ts_tablet_manager.cc:1367] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20240501 20:09:25.575819 23223 tablet_bootstrap.cc:492] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:25.576010 23223 ts_tablet_manager.cc:1367] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20240501 20:09:25.576093 23221 raft_consensus.cc:348] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.576337 23221 raft_consensus.cc:374] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.576413 23221 raft_consensus.cc:729] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.576656 23221 consensus_queue.cc:260] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.576808 23223 raft_consensus.cc:348] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.577093 23223 raft_consensus.cc:374] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.577214 23223 raft_consensus.cc:729] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.577306 23221 ts_tablet_manager.cc:1398] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:25.577510 23223 consensus_queue.cc:260] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.577832 23221 tablet_bootstrap.cc:492] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:25.579231 23223 ts_tablet_manager.cc:1398] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:25.579586 23221 tablet_bootstrap.cc:654] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:25.584643 23221 tablet_bootstrap.cc:492] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:25.584782 23221 ts_tablet_manager.cc:1367] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:09:25.585494 23221 raft_consensus.cc:348] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.585649 23221 raft_consensus.cc:374] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:25.585744 23221 raft_consensus.cc:729] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:25.585951 23221 consensus_queue.cc:260] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.586414 23221 ts_tablet_manager.cc:1398] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:25.601925 23075 raft_consensus.cc:482] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:25.602078 23075 raft_consensus.cc:504] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.602878 23075 leader_election.cc:288] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:25.603199 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:25.603365 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:25.603454 21228 raft_consensus.cc:2454] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:25.603641 20967 raft_consensus.cc:2454] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:25.603960 21294 leader_election.cc:302] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:25.604377 23075 raft_consensus.cc:2781] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:25.604501 23075 raft_consensus.cc:482] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:25.604604 23075 raft_consensus.cc:3037] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:25.607158 23075 raft_consensus.cc:504] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.607676 23075 leader_election.cc:288] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:25.608049 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:25.608258 21228 raft_consensus.cc:3037] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:25.608281 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:25.608482 20967 raft_consensus.cc:3037] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:25.610189 21228 raft_consensus.cc:2454] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:25.610412 20967 raft_consensus.cc:2454] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:25.610541 21294 leader_election.cc:302] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:25.610904 23117 raft_consensus.cc:2781] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:25.611397 23117 raft_consensus.cc:686] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:25.611726 23117 consensus_queue.cc:237] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.615044 20817 catalog_manager.cc:5484] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:25.792766 23117 raft_consensus.cc:482] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:25.792999 23117 raft_consensus.cc:504] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.793692 23117 leader_election.cc:288] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:25.794131 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d14c3c71012405e81dabba5f127433d" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:25.794183 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d14c3c71012405e81dabba5f127433d" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:25.794428 20965 raft_consensus.cc:2454] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:25.794461 21097 raft_consensus.cc:2454] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:25.794950 21291 leader_election.cc:302] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:25.795248 23117 raft_consensus.cc:2781] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:25.795387 23117 raft_consensus.cc:482] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:25.795521 23117 raft_consensus.cc:3037] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:25.797470 23117 raft_consensus.cc:504] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.798050 23117 leader_election.cc:288] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:25.798498 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d14c3c71012405e81dabba5f127433d" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:25.798513 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d14c3c71012405e81dabba5f127433d" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:25.798727 21097 raft_consensus.cc:3037] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:25.798727 20965 raft_consensus.cc:3037] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:25.801574 21097 raft_consensus.cc:2454] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:25.801575 20965 raft_consensus.cc:2454] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:25.801990 21291 leader_election.cc:302] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:25.802280 23117 raft_consensus.cc:2781] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:25.802438 23117 raft_consensus.cc:686] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:25.802672 23117 consensus_queue.cc:237] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:25.805773 20817 catalog_manager.cc:5484] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:25.953001 23117 raft_consensus.cc:482] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:25.953254 23117 raft_consensus.cc:504] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:25.954007 23117 leader_election.cc:288] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:25.954526 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748b4fd500f84a359bd8ee05b861300d" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:25.954756 21097 raft_consensus.cc:2454] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:25.955286 21291 leader_election.cc:302] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:25.955637 23117 raft_consensus.cc:2781] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:25.955755 23117 raft_consensus.cc:482] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:25.955842 23117 raft_consensus.cc:3037] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:25.956458 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748b4fd500f84a359bd8ee05b861300d" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:25.956671 21226 raft_consensus.cc:2454] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:25.958954 23117 raft_consensus.cc:504] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:25.959563 23117 leader_election.cc:288] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:25.959998 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748b4fd500f84a359bd8ee05b861300d" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:25.960232 21097 raft_consensus.cc:3037] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:25.960243 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748b4fd500f84a359bd8ee05b861300d" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:25.960445 21226 raft_consensus.cc:3037] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:25.963019 21097 raft_consensus.cc:2454] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:25.963157 21226 raft_consensus.cc:2454] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:25.963521 21291 leader_election.cc:302] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:25.963928 23117 raft_consensus.cc:2781] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:25.964087 23117 raft_consensus.cc:686] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:25.964349 23117 consensus_queue.cc:237] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:25.968487 20817 catalog_manager.cc:5484] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:26.061838 23117 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:26.078428 23117 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:26.199632 23117 consensus_queue.cc:1035] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:26.206651 23075 consensus_queue.cc:1035] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:26.381109 23075 consensus_queue.cc:1035] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:26.385202 23233 consensus_queue.cc:1035] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:26.445992 23242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:26.446468 23242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:26.458611 23242 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
INSERT report
rows total: 2000
time total: 233.194 ms
time per row: 0.116597 ms
I20240501 20:09:26.852764 20817 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:26.862962 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 99fb3a1b19924865b1c10fa129aa9192 (DEFAULT_TABLE table=rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:26.863176 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 99fb3a1b19924865b1c10fa129aa9192 (DEFAULT_TABLE table=rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:26.863590 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99fb3a1b19924865b1c10fa129aa9192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.863488 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 591bdca6da8849f4a6300b0db04d3754 (DEFAULT_TABLE table=rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:26.863628 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99fb3a1b19924865b1c10fa129aa9192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.864022 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591bdca6da8849f4a6300b0db04d3754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.869411 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 99fb3a1b19924865b1c10fa129aa9192 (DEFAULT_TABLE table=rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:26.869561 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 591bdca6da8849f4a6300b0db04d3754 (DEFAULT_TABLE table=rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:26.869916 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99fb3a1b19924865b1c10fa129aa9192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.870164 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591bdca6da8849f4a6300b0db04d3754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.870694 23295 tablet_bootstrap.cc:492] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:26.872512 23295 tablet_bootstrap.cc:654] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.876883 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 591bdca6da8849f4a6300b0db04d3754 (DEFAULT_TABLE table=rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:26.877357 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591bdca6da8849f4a6300b0db04d3754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.880206 23298 tablet_bootstrap.cc:492] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:26.880322 21338 tablet_service.cc:1467] Processing CreateTablet for tablet d61793ef4b84426c839c55e2fabfe1f6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:26.880499 23299 tablet_bootstrap.cc:492] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:26.880817 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61793ef4b84426c839c55e2fabfe1f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.882216 21207 tablet_service.cc:1467] Processing CreateTablet for tablet d61793ef4b84426c839c55e2fabfe1f6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:26.882632 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61793ef4b84426c839c55e2fabfe1f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.882752 20945 tablet_service.cc:1467] Processing CreateTablet for tablet d61793ef4b84426c839c55e2fabfe1f6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:26.883152 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61793ef4b84426c839c55e2fabfe1f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.883612 23296 tablet_bootstrap.cc:492] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:26.883160 23299 tablet_bootstrap.cc:654] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.887058 23296 tablet_bootstrap.cc:654] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.890719 23298 tablet_bootstrap.cc:654] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.891283 23296 tablet_bootstrap.cc:492] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:26.891490 23296 ts_tablet_manager.cc:1367] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:09:26.892486 23296 raft_consensus.cc:348] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:26.893374 23296 raft_consensus.cc:374] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.893570 23296 raft_consensus.cc:729] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.893999 23296 consensus_queue.cc:260] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:26.896358 23296 ts_tablet_manager.cc:1398] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:09:26.897642 23298 tablet_bootstrap.cc:492] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:26.897847 23298 ts_tablet_manager.cc:1367] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20240501 20:09:26.891227 23299 tablet_bootstrap.cc:492] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:26.898283 23299 ts_tablet_manager.cc:1367] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240501 20:09:26.898630 23298 raft_consensus.cc:348] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:26.898862 23298 raft_consensus.cc:374] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.898967 23298 raft_consensus.cc:729] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.899235 23298 consensus_queue.cc:260] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:26.899009 23299 raft_consensus.cc:348] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:26.899564 23299 raft_consensus.cc:374] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.899711 23299 raft_consensus.cc:729] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.900875 23298 ts_tablet_manager.cc:1398] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:26.901247 23298 tablet_bootstrap.cc:492] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:26.903136 23299 consensus_queue.cc:260] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:26.903606 23296 tablet_bootstrap.cc:492] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:26.903755 23298 tablet_bootstrap.cc:654] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.903749 23299 ts_tablet_manager.cc:1398] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:09:26.904881 23299 tablet_bootstrap.cc:492] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:26.905483 23296 tablet_bootstrap.cc:654] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.906368 23299 tablet_bootstrap.cc:654] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.907873 23295 tablet_bootstrap.cc:492] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:26.908084 23295 ts_tablet_manager.cc:1367] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.005s
I20240501 20:09:26.909934 23295 raft_consensus.cc:348] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:26.910924 23295 raft_consensus.cc:374] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.911976 23298 tablet_bootstrap.cc:492] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:26.912217 23298 ts_tablet_manager.cc:1367] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 20:09:26.913244 23298 raft_consensus.cc:348] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:26.913666 23298 raft_consensus.cc:374] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.913801 23298 raft_consensus.cc:729] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.914110 23298 consensus_queue.cc:260] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:26.915844 23298 ts_tablet_manager.cc:1398] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:26.918706 23295 raft_consensus.cc:729] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.919054 23295 consensus_queue.cc:260] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:26.919965 23295 ts_tablet_manager.cc:1398] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 20:09:26.920281 23295 tablet_bootstrap.cc:492] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:26.920703 23296 tablet_bootstrap.cc:492] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:26.920869 23296 ts_tablet_manager.cc:1367] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.002s
I20240501 20:09:26.921556 23296 raft_consensus.cc:348] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:26.921733 23296 raft_consensus.cc:374] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.921814 23296 raft_consensus.cc:729] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.922102 23295 tablet_bootstrap.cc:654] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.922056 23296 consensus_queue.cc:260] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:26.922636 23296 ts_tablet_manager.cc:1398] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:26.924108 23295 tablet_bootstrap.cc:492] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:26.924341 23295 ts_tablet_manager.cc:1367] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 20:09:26.924888 23299 tablet_bootstrap.cc:492] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:26.925122 23299 ts_tablet_manager.cc:1367] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.006s
I20240501 20:09:26.925168 23295 raft_consensus.cc:348] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:26.925433 23295 raft_consensus.cc:374] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.925544 23295 raft_consensus.cc:729] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.926880 23295 consensus_queue.cc:260] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:26.927489 23295 ts_tablet_manager.cc:1398] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:26.933789 23299 raft_consensus.cc:348] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:26.934034 23299 raft_consensus.cc:374] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.934139 23299 raft_consensus.cc:729] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.934391 23299 consensus_queue.cc:260] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:26.935101 23299 ts_tablet_manager.cc:1398] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240501 20:09:26.935461 23299 tablet_bootstrap.cc:492] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:26.937206 23299 tablet_bootstrap.cc:654] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.940974 23299 tablet_bootstrap.cc:492] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:26.941196 23299 ts_tablet_manager.cc:1367] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 20:09:26.941915 23299 raft_consensus.cc:348] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:26.942546 23299 raft_consensus.cc:374] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.942682 23299 raft_consensus.cc:729] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.942945 23299 consensus_queue.cc:260] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:26.943508 23299 ts_tablet_manager.cc:1398] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:27.010459 23148 raft_consensus.cc:482] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:27.010674 23148 raft_consensus.cc:504] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:27.011436 23148 leader_election.cc:288] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:27.011966 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61793ef4b84426c839c55e2fabfe1f6" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:27.012001 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61793ef4b84426c839c55e2fabfe1f6" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:27.012229 20967 raft_consensus.cc:2454] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:27.012235 21358 raft_consensus.cc:2454] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:27.012789 21161 leader_election.cc:302] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:27.013140 23148 raft_consensus.cc:2781] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:27.013235 23148 raft_consensus.cc:482] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:27.013332 23148 raft_consensus.cc:3037] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:27.015440 23148 raft_consensus.cc:504] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:27.016055 23148 leader_election.cc:288] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:27.016543 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61793ef4b84426c839c55e2fabfe1f6" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:27.016561 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61793ef4b84426c839c55e2fabfe1f6" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:27.016739 20967 raft_consensus.cc:3037] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:27.016737 21358 raft_consensus.cc:3037] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:27.018572 21358 raft_consensus.cc:2454] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:27.018956 20967 raft_consensus.cc:2454] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:27.018990 21162 leader_election.cc:302] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:27.019284 23148 raft_consensus.cc:2781] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:27.019694 23148 raft_consensus.cc:686] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:27.019973 23148 consensus_queue.cc:237] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:27.022977 20817 catalog_manager.cc:5484] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:27.024523 23108 raft_consensus.cc:482] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:27.024658 23108 raft_consensus.cc:504] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:27.025344 23108 leader_election.cc:288] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:27.025877 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591bdca6da8849f4a6300b0db04d3754" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:27.025991 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591bdca6da8849f4a6300b0db04d3754" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:27.026063 21226 raft_consensus.cc:2454] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:27.026234 21097 raft_consensus.cc:2454] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:27.026495 20903 leader_election.cc:302] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:27.026805 23108 raft_consensus.cc:2781] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:27.026904 23108 raft_consensus.cc:482] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:27.026999 23108 raft_consensus.cc:3037] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:27.029011 23108 raft_consensus.cc:504] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:27.029546 23108 leader_election.cc:288] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:27.030009 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591bdca6da8849f4a6300b0db04d3754" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:27.030067 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591bdca6da8849f4a6300b0db04d3754" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:27.030220 21226 raft_consensus.cc:3037] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:27.030298 21097 raft_consensus.cc:3037] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:27.032570 21097 raft_consensus.cc:2454] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:27.032608 21226 raft_consensus.cc:2454] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:27.033030 20900 leader_election.cc:302] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:27.033342 23215 raft_consensus.cc:2781] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:27.033478 23215 raft_consensus.cc:686] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:27.033762 23215 consensus_queue.cc:237] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:27.037675 20823 catalog_manager.cc:5484] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:27.187829 23215 raft_consensus.cc:482] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:27.188028 23215 raft_consensus.cc:504] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:27.188560 23215 leader_election.cc:288] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:27.189002 21096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fb3a1b19924865b1c10fa129aa9192" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:27.189044 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fb3a1b19924865b1c10fa129aa9192" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:27.189186 21096 raft_consensus.cc:2454] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:27.189282 21358 raft_consensus.cc:2454] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:27.189584 20900 leader_election.cc:302] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:27.189975 23215 raft_consensus.cc:2781] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:27.190053 23215 raft_consensus.cc:482] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:27.190155 23215 raft_consensus.cc:3037] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:27.192067 23215 raft_consensus.cc:504] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:27.192606 23215 leader_election.cc:288] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:27.193009 21096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fb3a1b19924865b1c10fa129aa9192" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:27.193099 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fb3a1b19924865b1c10fa129aa9192" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:27.193212 21096 raft_consensus.cc:3037] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:27.193306 21358 raft_consensus.cc:3037] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:27.196136 21096 raft_consensus.cc:2454] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:27.196134 21358 raft_consensus.cc:2454] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:27.196532 20901 leader_election.cc:302] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:27.196791 23215 raft_consensus.cc:2781] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:27.196982 23215 raft_consensus.cc:686] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:27.197270 23215 consensus_queue.cc:237] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:27.200603 20823 catalog_manager.cc:5484] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:27.448139 23215 consensus_queue.cc:1035] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:27.472374 23108 consensus_queue.cc:1035] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:27.586836 23148 consensus_queue.cc:1035] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:27.591279 23148 consensus_queue.cc:1035] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:27.613304 23215 consensus_queue.cc:1035] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:27.627066 23215 consensus_queue.cc:1035] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:27.634657 23313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:27.635021 23313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:27.642889 23313 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
W20240501 20:09:27.943625 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 81529.9us
INSERT report
rows total: 2000
time total: 288.616 ms
time per row: 0.144308 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:29.076717 20822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:29.087924 21338 tablet_service.cc:1467] Processing CreateTablet for tablet e1476e8f4c9343928f68365fd2772f0f (DEFAULT_TABLE table=rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.088351 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1476e8f4c9343928f68365fd2772f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.088245 21207 tablet_service.cc:1467] Processing CreateTablet for tablet e1476e8f4c9343928f68365fd2772f0f (DEFAULT_TABLE table=rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.088673 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1476e8f4c9343928f68365fd2772f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.089280 21076 tablet_service.cc:1467] Processing CreateTablet for tablet e1476e8f4c9343928f68365fd2772f0f (DEFAULT_TABLE table=rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.089684 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1476e8f4c9343928f68365fd2772f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.091868 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7e956aedc44f56b3bf629c80ff65e4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.092154 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7e956aedc44f56b3bf629c80ff65e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.094710 23386 tablet_bootstrap.cc:492] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:29.095968 20945 tablet_service.cc:1467] Processing CreateTablet for tablet ca03186ff8b342dd958b205125b14177 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.096189 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7e956aedc44f56b3bf629c80ff65e4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.096328 23386 tablet_bootstrap.cc:654] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.096380 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca03186ff8b342dd958b205125b14177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.096581 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7e956aedc44f56b3bf629c80ff65e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.097836 21078 tablet_service.cc:1467] Processing CreateTablet for tablet ca03186ff8b342dd958b205125b14177 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.098188 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca03186ff8b342dd958b205125b14177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.103286 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7e956aedc44f56b3bf629c80ff65e4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.103430 21336 tablet_service.cc:1467] Processing CreateTablet for tablet ca03186ff8b342dd958b205125b14177 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.103749 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7e956aedc44f56b3bf629c80ff65e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.104007 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca03186ff8b342dd958b205125b14177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.114742 23388 tablet_bootstrap.cc:492] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:29.118288 23388 tablet_bootstrap.cc:654] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.118822 23386 tablet_bootstrap.cc:492] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:29.118981 23386 ts_tablet_manager.cc:1367] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.012s
I20240501 20:09:29.119881 23386 raft_consensus.cc:348] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:29.120098 23386 raft_consensus.cc:374] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.120226 23386 raft_consensus.cc:729] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.120442 23386 consensus_queue.cc:260] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:29.120726 23388 tablet_bootstrap.cc:492] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:29.121062 23388 ts_tablet_manager.cc:1367] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240501 20:09:29.121933 23388 raft_consensus.cc:348] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.122293 23388 raft_consensus.cc:374] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.122440 23388 raft_consensus.cc:729] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.122745 23388 consensus_queue.cc:260] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.123935 23388 ts_tablet_manager.cc:1398] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:29.124401 23388 tablet_bootstrap.cc:492] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:29.125118 23386 ts_tablet_manager.cc:1398] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:29.126089 23390 tablet_bootstrap.cc:492] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:29.126368 23388 tablet_bootstrap.cc:654] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.128613 23390 tablet_bootstrap.cc:654] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.129237 23389 tablet_bootstrap.cc:492] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:29.130352 23386 tablet_bootstrap.cc:492] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:29.131531 23386 tablet_bootstrap.cc:654] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.131636 23389 tablet_bootstrap.cc:654] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.134991 23388 tablet_bootstrap.cc:492] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:29.135293 23388 ts_tablet_manager.cc:1367] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 20:09:29.136336 23388 raft_consensus.cc:348] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:29.136653 23388 raft_consensus.cc:374] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.136754 23388 raft_consensus.cc:729] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.136821 23386 tablet_bootstrap.cc:492] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:29.137017 23386 ts_tablet_manager.cc:1367] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:09:29.137049 23388 consensus_queue.cc:260] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:29.137816 23388 ts_tablet_manager.cc:1398] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:29.138590 23386 raft_consensus.cc:348] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.138815 23386 raft_consensus.cc:374] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.138896 23386 raft_consensus.cc:729] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.139204 23390 tablet_bootstrap.cc:492] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:29.139127 23386 consensus_queue.cc:260] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.139357 23390 ts_tablet_manager.cc:1367] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240501 20:09:29.140112 23390 raft_consensus.cc:348] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:29.140275 23390 raft_consensus.cc:374] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.140400 23390 raft_consensus.cc:729] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.140612 23390 consensus_queue.cc:260] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:29.141045 23390 ts_tablet_manager.cc:1398] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:29.141371 23390 tablet_bootstrap.cc:492] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:29.141974 23386 ts_tablet_manager.cc:1398] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:29.142341 23386 tablet_bootstrap.cc:492] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:29.143299 23390 tablet_bootstrap.cc:654] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.143918 23386 tablet_bootstrap.cc:654] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.145673 23389 tablet_bootstrap.cc:492] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:29.145951 23389 ts_tablet_manager.cc:1367] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20240501 20:09:29.146883 23389 raft_consensus.cc:348] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.147265 23389 raft_consensus.cc:374] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.147428 23389 raft_consensus.cc:729] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.147760 23389 consensus_queue.cc:260] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.148447 23389 ts_tablet_manager.cc:1398] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:29.148701 23390 tablet_bootstrap.cc:492] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:29.148862 23390 ts_tablet_manager.cc:1367] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 20:09:29.149513 23390 raft_consensus.cc:348] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.149690 23390 raft_consensus.cc:374] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.149767 23390 raft_consensus.cc:729] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.150022 23390 consensus_queue.cc:260] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.150496 23390 ts_tablet_manager.cc:1398] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:29.150754 23390 tablet_bootstrap.cc:492] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:29.152558 23390 tablet_bootstrap.cc:654] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.154080 23390 tablet_bootstrap.cc:492] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:29.154211 23390 ts_tablet_manager.cc:1367] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:29.154738 23390 raft_consensus.cc:348] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.154887 23390 raft_consensus.cc:374] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.155016 23390 raft_consensus.cc:729] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.155210 23390 consensus_queue.cc:260] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.155692 23390 ts_tablet_manager.cc:1398] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:29.162665 23386 tablet_bootstrap.cc:492] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:29.162839 23386 ts_tablet_manager.cc:1367] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.003s
I20240501 20:09:29.163529 23386 raft_consensus.cc:348] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.163726 23386 raft_consensus.cc:374] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.163807 23386 raft_consensus.cc:729] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.164033 23386 consensus_queue.cc:260] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.164538 23386 ts_tablet_manager.cc:1398] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:29.195659 23380 raft_consensus.cc:482] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:29.195819 23380 raft_consensus.cc:504] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:29.196457 23380 leader_election.cc:288] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:29.196830 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1476e8f4c9343928f68365fd2772f0f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:29.196893 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1476e8f4c9343928f68365fd2772f0f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:29.197037 21358 raft_consensus.cc:2454] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:29.197160 21228 raft_consensus.cc:2454] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:29.197654 21032 leader_election.cc:302] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:29.197947 23380 raft_consensus.cc:2781] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:29.198069 23380 raft_consensus.cc:482] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:29.198161 23380 raft_consensus.cc:3037] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.200770 23380 raft_consensus.cc:504] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:29.201622 23380 leader_election.cc:288] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:29.201733 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1476e8f4c9343928f68365fd2772f0f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:29.201967 21358 raft_consensus.cc:3037] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.201927 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1476e8f4c9343928f68365fd2772f0f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:29.202147 21228 raft_consensus.cc:3037] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.203888 21358 raft_consensus.cc:2454] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:29.203997 21228 raft_consensus.cc:2454] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:29.204241 21032 leader_election.cc:302] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:29.204475 23381 raft_consensus.cc:482] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:29.204530 23380 raft_consensus.cc:2781] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:29.204702 23380 raft_consensus.cc:686] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:29.204615 23381 raft_consensus.cc:504] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.205060 23380 consensus_queue.cc:237] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:29.205310 23381 leader_election.cc:288] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:29.205724 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca03186ff8b342dd958b205125b14177" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:29.205736 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca03186ff8b342dd958b205125b14177" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:29.205922 21097 raft_consensus.cc:2454] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:29.205962 20966 raft_consensus.cc:2454] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:29.206315 21291 leader_election.cc:302] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:29.206612 23381 raft_consensus.cc:2781] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:29.206758 23381 raft_consensus.cc:482] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:29.206864 23381 raft_consensus.cc:3037] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.210589 23381 raft_consensus.cc:504] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.211474 23381 leader_election.cc:288] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:29.211758 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca03186ff8b342dd958b205125b14177" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:29.211875 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca03186ff8b342dd958b205125b14177" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:29.211988 20966 raft_consensus.cc:3037] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.212054 21097 raft_consensus.cc:3037] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.215246 20966 raft_consensus.cc:2454] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:29.215631 21291 leader_election.cc:302] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:29.215967 23381 raft_consensus.cc:2781] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:29.216199 23381 raft_consensus.cc:686] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:29.216459 23381 consensus_queue.cc:237] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.216651 21097 raft_consensus.cc:2454] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:29.216717 20824 catalog_manager.cc:5484] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:29.220095 20823 catalog_manager.cc:5484] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:29.306188 23380 raft_consensus.cc:482] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:29.306406 23380 raft_consensus.cc:504] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.306988 23380 leader_election.cc:288] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:29.307420 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e956aedc44f56b3bf629c80ff65e4" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:29.307504 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e956aedc44f56b3bf629c80ff65e4" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:29.307684 21228 raft_consensus.cc:2454] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:29.307732 21358 raft_consensus.cc:2454] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:29.308140 21032 leader_election.cc:302] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:29.308458 23380 raft_consensus.cc:2781] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:29.308543 23380 raft_consensus.cc:482] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:29.308651 23380 raft_consensus.cc:3037] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.310954 23380 raft_consensus.cc:504] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.311899 23380 leader_election.cc:288] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:29.312270 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e956aedc44f56b3bf629c80ff65e4" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:29.312304 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e956aedc44f56b3bf629c80ff65e4" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:29.312428 21358 raft_consensus.cc:3037] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.312484 21228 raft_consensus.cc:3037] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.314347 21228 raft_consensus.cc:2454] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:29.314409 21358 raft_consensus.cc:2454] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:29.314733 21034 leader_election.cc:302] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:29.314983 23380 raft_consensus.cc:2781] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:29.315174 23380 raft_consensus.cc:686] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:29.315428 23380 consensus_queue.cc:237] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:29.319160 20823 catalog_manager.cc:5484] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:29.626721 23164 consensus_queue.cc:1035] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:29.632630 23164 consensus_queue.cc:1035] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:29.688972 23381 consensus_queue.cc:1035] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:29.695753 23381 consensus_queue.cc:1035] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:29.726840 23402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:29.727197 23402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:29.735028 23402 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:29.764225 23164 consensus_queue.cc:1035] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
I20240501 20:09:29.769255 23380 consensus_queue.cc:1035] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 271.767 ms
time per row: 0.135884 ms
I20240501 20:09:30.172082 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:30.180769 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 09a3d74fea894fb6b0e7fc58257ca56a (DEFAULT_TABLE table=rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.181231 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a3d74fea894fb6b0e7fc58257ca56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.181402 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 09a3d74fea894fb6b0e7fc58257ca56a (DEFAULT_TABLE table=rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.181710 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a3d74fea894fb6b0e7fc58257ca56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.181672 21338 tablet_service.cc:1467] Processing CreateTablet for tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.182060 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.186201 20946 tablet_service.cc:1467] Processing CreateTablet for tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.186558 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.187358 21078 tablet_service.cc:1467] Processing CreateTablet for tablet a396d90619784ba0ab865e6d07237db7 (DEFAULT_TABLE table=rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.187721 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a396d90619784ba0ab865e6d07237db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.188112 20947 tablet_service.cc:1467] Processing CreateTablet for tablet a396d90619784ba0ab865e6d07237db7 (DEFAULT_TABLE table=rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.188462 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a396d90619784ba0ab865e6d07237db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.191859 21206 tablet_service.cc:1467] Processing CreateTablet for tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.192314 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 09a3d74fea894fb6b0e7fc58257ca56a (DEFAULT_TABLE table=rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.192656 21208 tablet_service.cc:1467] Processing CreateTablet for tablet a396d90619784ba0ab865e6d07237db7 (DEFAULT_TABLE table=rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:30.192314 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.193832 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a396d90619784ba0ab865e6d07237db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.194108 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a3d74fea894fb6b0e7fc58257ca56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:30.209460 23470 tablet_bootstrap.cc:492] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:30.209699 23471 tablet_bootstrap.cc:492] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:30.211428 23470 tablet_bootstrap.cc:654] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.211767 23468 tablet_bootstrap.cc:492] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:30.212014 23469 tablet_bootstrap.cc:492] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:30.212977 23471 tablet_bootstrap.cc:654] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.213735 23468 tablet_bootstrap.cc:654] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.213738 23469 tablet_bootstrap.cc:654] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.214857 23471 tablet_bootstrap.cc:492] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:30.214993 23471 ts_tablet_manager.cc:1367] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:09:30.215837 23471 raft_consensus.cc:348] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:30.216176 23471 raft_consensus.cc:374] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.216321 23471 raft_consensus.cc:729] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.216518 23469 tablet_bootstrap.cc:492] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:30.216681 23469 ts_tablet_manager.cc:1367] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:30.216693 23471 consensus_queue.cc:260] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:30.217423 23471 ts_tablet_manager.cc:1398] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:30.217504 23469 raft_consensus.cc:348] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:30.217713 23469 raft_consensus.cc:374] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.217816 23469 raft_consensus.cc:729] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.218063 23469 consensus_queue.cc:260] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:30.218669 23469 ts_tablet_manager.cc:1398] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:30.220430 23469 tablet_bootstrap.cc:492] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:30.221882 23470 tablet_bootstrap.cc:492] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:30.222086 23470 ts_tablet_manager.cc:1367] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20240501 20:09:30.222368 23469 tablet_bootstrap.cc:654] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.223033 23470 raft_consensus.cc:348] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:30.223347 23470 raft_consensus.cc:374] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.223487 23470 raft_consensus.cc:729] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.223830 23470 consensus_queue.cc:260] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:30.225872 23470 ts_tablet_manager.cc:1398] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:30.226200 23470 tablet_bootstrap.cc:492] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:30.228029 23470 tablet_bootstrap.cc:654] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.233533 23468 tablet_bootstrap.cc:492] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:30.233798 23468 ts_tablet_manager.cc:1367] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.007s
I20240501 20:09:30.235193 23468 raft_consensus.cc:348] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:30.235476 23468 raft_consensus.cc:374] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.235630 23468 raft_consensus.cc:729] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.235922 23468 consensus_queue.cc:260] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:30.236771 23468 ts_tablet_manager.cc:1398] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:30.237200 23468 tablet_bootstrap.cc:492] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:30.241667 23468 tablet_bootstrap.cc:654] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.251456 23470 tablet_bootstrap.cc:492] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:30.251698 23470 ts_tablet_manager.cc:1367] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.004s
I20240501 20:09:30.251883 23469 tablet_bootstrap.cc:492] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:30.252071 23469 ts_tablet_manager.cc:1367] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.024s
I20240501 20:09:30.252182 23468 tablet_bootstrap.cc:492] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:30.252352 23468 ts_tablet_manager.cc:1367] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 20:09:30.252535 23470 raft_consensus.cc:348] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:30.252786 23470 raft_consensus.cc:374] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.252772 23469 raft_consensus.cc:348] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:30.252893 23470 raft_consensus.cc:729] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.252996 23469 raft_consensus.cc:374] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.253079 23469 raft_consensus.cc:729] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.253120 23468 raft_consensus.cc:348] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:30.253178 23470 consensus_queue.cc:260] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:30.253336 23468 raft_consensus.cc:374] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.253320 23469 consensus_queue.cc:260] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:30.253443 23468 raft_consensus.cc:729] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.253827 23468 consensus_queue.cc:260] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:30.253957 23469 ts_tablet_manager.cc:1398] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:30.254418 23468 ts_tablet_manager.cc:1398] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:30.254475 23469 tablet_bootstrap.cc:492] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:30.254839 23470 ts_tablet_manager.cc:1398] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:30.255111 23470 tablet_bootstrap.cc:492] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:30.256100 23469 tablet_bootstrap.cc:654] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.256515 23470 tablet_bootstrap.cc:654] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:30.261888 23469 tablet_bootstrap.cc:492] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:30.262120 23469 ts_tablet_manager.cc:1367] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20240501 20:09:30.262383 23470 tablet_bootstrap.cc:492] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:30.262584 23470 ts_tablet_manager.cc:1367] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 20:09:30.262930 23469 raft_consensus.cc:348] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:30.263224 23469 raft_consensus.cc:374] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.263362 23469 raft_consensus.cc:729] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.263382 23470 raft_consensus.cc:348] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:30.263675 23470 raft_consensus.cc:374] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:30.263783 23470 raft_consensus.cc:729] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:30.263681 23469 consensus_queue.cc:260] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:30.264335 23469 ts_tablet_manager.cc:1398] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:30.264297 23470 consensus_queue.cc:260] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:30.264887 23470 ts_tablet_manager.cc:1398] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:30.265165 23381 raft_consensus.cc:482] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:30.265343 23381 raft_consensus.cc:504] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:30.266083 23381 leader_election.cc:288] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:30.266474 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fd6fe82bda4d14b3efe2fc2ec0ddc2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:30.266705 20966 raft_consensus.cc:2454] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:30.266680 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fd6fe82bda4d14b3efe2fc2ec0ddc2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:30.266872 21228 raft_consensus.cc:2454] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:30.267153 21291 leader_election.cc:302] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:30.267457 23381 raft_consensus.cc:2781] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:30.267532 23381 raft_consensus.cc:482] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:30.267658 23381 raft_consensus.cc:3037] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.269874 23381 raft_consensus.cc:504] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:30.270450 23381 leader_election.cc:288] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:30.270781 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fd6fe82bda4d14b3efe2fc2ec0ddc2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:30.270912 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fd6fe82bda4d14b3efe2fc2ec0ddc2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:30.270982 20966 raft_consensus.cc:3037] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.271107 21228 raft_consensus.cc:3037] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.273811 21228 raft_consensus.cc:2454] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:30.273841 20966 raft_consensus.cc:2454] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:30.274276 21291 leader_election.cc:302] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:30.274578 23381 raft_consensus.cc:2781] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:30.274773 23381 raft_consensus.cc:686] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:30.275040 23381 consensus_queue.cc:237] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:30.278263 20820 catalog_manager.cc:5484] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:30.460500 23403 raft_consensus.cc:482] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:30.460669 23403 raft_consensus.cc:504] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:30.461308 23403 leader_election.cc:288] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:30.461715 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a396d90619784ba0ab865e6d07237db7" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:30.461783 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a396d90619784ba0ab865e6d07237db7" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:30.461962 21226 raft_consensus.cc:2454] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:30.462031 21097 raft_consensus.cc:2454] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:30.462332 20903 leader_election.cc:302] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:30.462575 23403 raft_consensus.cc:2781] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:30.462733 23403 raft_consensus.cc:482] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:30.462826 23403 raft_consensus.cc:3037] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.465483 23403 raft_consensus.cc:504] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:30.466372 23403 leader_election.cc:288] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:30.466524 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a396d90619784ba0ab865e6d07237db7" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:30.466739 21226 raft_consensus.cc:3037] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.466873 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a396d90619784ba0ab865e6d07237db7" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:30.467116 21097 raft_consensus.cc:3037] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.469980 21226 raft_consensus.cc:2454] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:30.470628 20903 leader_election.cc:302] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:30.471228 23364 raft_consensus.cc:2781] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:30.471344 23364 raft_consensus.cc:686] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:30.471642 23364 consensus_queue.cc:237] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:30.472443 21097 raft_consensus.cc:2454] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:30.481038 23380 raft_consensus.cc:482] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:30.481334 23380 raft_consensus.cc:504] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:30.482440 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a3d74fea894fb6b0e7fc58257ca56a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:30.482693 21226 raft_consensus.cc:2454] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:30.483147 21034 leader_election.cc:302] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:30.483407 23380 leader_election.cc:288] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:30.483451 23455 raft_consensus.cc:2781] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:30.483573 23455 raft_consensus.cc:482] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:30.483719 23455 raft_consensus.cc:3037] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.483852 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a3d74fea894fb6b0e7fc58257ca56a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:30.484081 20966 raft_consensus.cc:2454] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:30.484385 20820 catalog_manager.cc:5484] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:30.487349 23455 raft_consensus.cc:504] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:30.489055 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a3d74fea894fb6b0e7fc58257ca56a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:30.489183 23455 leader_election.cc:288] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:30.489478 20966 raft_consensus.cc:3037] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.489759 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a3d74fea894fb6b0e7fc58257ca56a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:30.489970 21226 raft_consensus.cc:3037] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.492790 21226 raft_consensus.cc:2454] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:30.493188 20966 raft_consensus.cc:2454] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:30.493239 21034 leader_election.cc:302] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:30.493538 23455 raft_consensus.cc:2781] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:30.493695 23455 raft_consensus.cc:686] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:30.493999 23455 consensus_queue.cc:237] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:30.497323 20820 catalog_manager.cc:5484] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:30.654424 23381 consensus_queue.cc:1035] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:30.673293 23491 consensus_queue.cc:1035] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:30.865978 23486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:30.866376 23486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:30.874876 23486 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:30.883615 23403 consensus_queue.cc:1035] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:30.889705 23484 consensus_queue.cc:1035] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
I20240501 20:09:31.067574 23380 consensus_queue.cc:1035] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:31.099455 23455 consensus_queue.cc:1035] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 272.995 ms
time per row: 0.136497 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:32.315578 20824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:32.324705 21338 tablet_service.cc:1467] Processing CreateTablet for tablet 75a1b78bb9574b588e6bf69ae7e68bc3 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.325203 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a1b78bb9574b588e6bf69ae7e68bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.325137 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 75a1b78bb9574b588e6bf69ae7e68bc3 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.325603 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a1b78bb9574b588e6bf69ae7e68bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.327150 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2f74307a194dc49d184d4ea288b36d (DEFAULT_TABLE table=rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.327174 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 8d75b25758954877ab36e8029fb29463 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.327674 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2f74307a194dc49d184d4ea288b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.327766 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2f74307a194dc49d184d4ea288b36d (DEFAULT_TABLE table=rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.328114 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2f74307a194dc49d184d4ea288b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.331722 21078 tablet_service.cc:1467] Processing CreateTablet for tablet 8d75b25758954877ab36e8029fb29463 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.332111 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d75b25758954877ab36e8029fb29463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.332120 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d75b25758954877ab36e8029fb29463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.334193 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 75a1b78bb9574b588e6bf69ae7e68bc3 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.334677 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a1b78bb9574b588e6bf69ae7e68bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.336702 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2f74307a194dc49d184d4ea288b36d (DEFAULT_TABLE table=rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.337109 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2f74307a194dc49d184d4ea288b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.341464 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 8d75b25758954877ab36e8029fb29463 (DEFAULT_TABLE table=rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.341883 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d75b25758954877ab36e8029fb29463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.349133 23550 tablet_bootstrap.cc:492] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:32.350049 23552 tablet_bootstrap.cc:492] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:32.351068 23550 tablet_bootstrap.cc:654] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.352452 23553 tablet_bootstrap.cc:492] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:32.353188 23552 tablet_bootstrap.cc:654] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.353542 23551 tablet_bootstrap.cc:492] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:32.354161 23553 tablet_bootstrap.cc:654] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.355038 23551 tablet_bootstrap.cc:654] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.355868 23552 tablet_bootstrap.cc:492] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:32.356076 23552 ts_tablet_manager.cc:1367] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:09:32.356868 23553 tablet_bootstrap.cc:492] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:32.357013 23550 tablet_bootstrap.cc:492] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:32.357028 23553 ts_tablet_manager.cc:1367] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 20:09:32.356922 23552 raft_consensus.cc:348] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.357168 23550 ts_tablet_manager.cc:1367] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:09:32.357262 23552 raft_consensus.cc:374] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.357352 23552 raft_consensus.cc:729] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.357590 23552 consensus_queue.cc:260] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.357762 23553 raft_consensus.cc:348] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:32.357985 23551 tablet_bootstrap.cc:492] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:32.357978 23553 raft_consensus.cc:374] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.358088 23553 raft_consensus.cc:729] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.358018 23550 raft_consensus.cc:348] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:32.358165 23551 ts_tablet_manager.cc:1367] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:32.358227 23550 raft_consensus.cc:374] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.358312 23550 raft_consensus.cc:729] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.358284 23553 consensus_queue.cc:260] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:32.358527 23550 consensus_queue.cc:260] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:32.358798 23553 ts_tablet_manager.cc:1398] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:32.358875 23551 raft_consensus.cc:348] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:32.359067 23553 tablet_bootstrap.cc:492] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:32.359059 23550 ts_tablet_manager.cc:1398] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:32.359095 23551 raft_consensus.cc:374] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.359241 23551 raft_consensus.cc:729] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.359347 23550 tablet_bootstrap.cc:492] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:32.359572 23552 ts_tablet_manager.cc:1398] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:32.359483 23551 consensus_queue.cc:260] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:32.359829 23552 tablet_bootstrap.cc:492] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:32.360035 23551 ts_tablet_manager.cc:1398] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:32.360308 23551 tablet_bootstrap.cc:492] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:32.361945 23551 tablet_bootstrap.cc:654] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.363710 23482 raft_consensus.cc:482] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.363869 23482 raft_consensus.cc:504] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.364599 23482 leader_election.cc:288] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:32.365334 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f74307a194dc49d184d4ea288b36d" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:32.366355 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f74307a194dc49d184d4ea288b36d" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:32.366528 23553 tablet_bootstrap.cc:654] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
W20240501 20:09:32.367228 21161 leader_election.cc:341] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:32.368880 23550 tablet_bootstrap.cc:654] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
W20240501 20:09:32.369346 21162 leader_election.cc:341] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:32.369493 21162 leader_election.cc:302] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20; no voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0
I20240501 20:09:32.370039 23553 tablet_bootstrap.cc:492] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:32.370039 23482 raft_consensus.cc:2726] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:32.370209 23553 ts_tablet_manager.cc:1367] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 20:09:32.371017 23553 raft_consensus.cc:348] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:32.371241 23553 raft_consensus.cc:374] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.371323 23553 raft_consensus.cc:729] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.371611 23553 consensus_queue.cc:260] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:32.372125 23553 ts_tablet_manager.cc:1398] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:32.372301 23552 tablet_bootstrap.cc:654] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.376331 23550 tablet_bootstrap.cc:492] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:32.376520 23550 ts_tablet_manager.cc:1367] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20240501 20:09:32.376803 23552 tablet_bootstrap.cc:492] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:32.377029 23552 ts_tablet_manager.cc:1367] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240501 20:09:32.377099 23551 tablet_bootstrap.cc:492] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:32.377298 23551 ts_tablet_manager.cc:1367] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20240501 20:09:32.377280 23550 raft_consensus.cc:348] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:32.377564 23550 raft_consensus.cc:374] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.377676 23550 raft_consensus.cc:729] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.377944 23550 consensus_queue.cc:260] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:32.378122 23551 raft_consensus.cc:348] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.378255 23552 raft_consensus.cc:348] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:32.378528 23552 raft_consensus.cc:374] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.378595 23550 ts_tablet_manager.cc:1398] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:32.378688 23552 raft_consensus.cc:729] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.378958 23550 tablet_bootstrap.cc:492] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:32.379042 23552 consensus_queue.cc:260] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:32.379868 23551 raft_consensus.cc:374] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.380023 23551 raft_consensus.cc:729] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.380311 23551 consensus_queue.cc:260] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.380851 23550 tablet_bootstrap.cc:654] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.380936 23551 ts_tablet_manager.cc:1398] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:32.382687 23552 ts_tablet_manager.cc:1398] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:32.382946 23550 tablet_bootstrap.cc:492] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:32.383085 23550 ts_tablet_manager.cc:1367] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:32.383843 23550 raft_consensus.cc:348] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.384064 23550 raft_consensus.cc:374] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.384173 23550 raft_consensus.cc:729] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.384388 23550 consensus_queue.cc:260] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.386601 23550 ts_tablet_manager.cc:1398] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:32.392153 23484 raft_consensus.cc:482] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.392342 23484 raft_consensus.cc:504] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:32.393137 23484 leader_election.cc:288] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:32.393469 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a1b78bb9574b588e6bf69ae7e68bc3" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:32.393731 21358 raft_consensus.cc:2454] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:32.394488 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a1b78bb9574b588e6bf69ae7e68bc3" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:32.394706 21097 raft_consensus.cc:2454] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:32.395184 20900 leader_election.cc:302] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:32.395507 23484 raft_consensus.cc:2781] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:32.395637 23484 raft_consensus.cc:482] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:32.395747 23484 raft_consensus.cc:3037] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.398332 23484 raft_consensus.cc:504] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:32.399086 23484 leader_election.cc:288] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:32.399600 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a1b78bb9574b588e6bf69ae7e68bc3" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:32.399821 21358 raft_consensus.cc:3037] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.400060 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a1b78bb9574b588e6bf69ae7e68bc3" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:32.400282 21097 raft_consensus.cc:3037] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.401803 21358 raft_consensus.cc:2454] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:32.402181 20901 leader_election.cc:302] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:32.402343 21097 raft_consensus.cc:2454] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:32.402465 23484 raft_consensus.cc:2781] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:32.402631 23484 raft_consensus.cc:686] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:32.402949 23484 consensus_queue.cc:237] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:32.406041 20821 catalog_manager.cc:5484] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:32.463522 23482 raft_consensus.cc:482] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.463732 23482 raft_consensus.cc:504] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:32.464432 23482 leader_election.cc:288] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:32.464843 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d75b25758954877ab36e8029fb29463" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:32.464951 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d75b25758954877ab36e8029fb29463" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:32.465180 21097 raft_consensus.cc:2454] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:32.465209 21358 raft_consensus.cc:2454] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:32.465552 21161 leader_election.cc:302] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:32.465808 23482 raft_consensus.cc:2781] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:32.465936 23482 raft_consensus.cc:482] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:32.466035 23482 raft_consensus.cc:3037] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.468385 23482 raft_consensus.cc:504] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:32.469095 23482 leader_election.cc:288] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:32.469394 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d75b25758954877ab36e8029fb29463" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:32.469621 21097 raft_consensus.cc:3037] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.469899 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d75b25758954877ab36e8029fb29463" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:32.470121 21358 raft_consensus.cc:3037] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.472383 21097 raft_consensus.cc:2454] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:32.472724 21161 leader_election.cc:302] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:32.472774 21358 raft_consensus.cc:2454] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:32.473026 23482 raft_consensus.cc:2781] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:32.473201 23482 raft_consensus.cc:686] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:32.473507 23482 consensus_queue.cc:237] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:32.477186 20821 catalog_manager.cc:5484] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:32.609535 23484 raft_consensus.cc:482] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.609717 23484 raft_consensus.cc:504] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.610345 23484 leader_election.cc:288] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:32.610762 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f74307a194dc49d184d4ea288b36d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:32.610908 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f74307a194dc49d184d4ea288b36d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:32.610989 21358 raft_consensus.cc:2454] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:32.611148 21228 raft_consensus.cc:2454] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:32.611421 20901 leader_election.cc:302] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:32.611726 23484 raft_consensus.cc:2781] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:32.611847 23484 raft_consensus.cc:482] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:32.611924 23484 raft_consensus.cc:3037] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.614014 23484 raft_consensus.cc:504] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.614622 23484 leader_election.cc:288] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:32.614907 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f74307a194dc49d184d4ea288b36d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:32.615093 21358 raft_consensus.cc:3037] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.615204 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2f74307a194dc49d184d4ea288b36d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:32.615382 21228 raft_consensus.cc:3037] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.617050 21358 raft_consensus.cc:2454] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:32.617355 21228 raft_consensus.cc:2454] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:32.617399 20901 leader_election.cc:302] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:32.617723 23484 raft_consensus.cc:2781] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:32.617875 23484 raft_consensus.cc:686] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:32.618139 23484 consensus_queue.cc:237] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:32.621457 20821 catalog_manager.cc:5484] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:32.850785 23482 consensus_queue.cc:1035] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:32.860078 23482 consensus_queue.cc:1035] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:32.933120 23545 consensus_queue.cc:1035] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:32.941458 23484 consensus_queue.cc:1035] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:32.999644 23567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:33.000013 23567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:33.007860 23567 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 20:09:33.072714 23484 consensus_queue.cc:1035] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:33.096837 23484 consensus_queue.cc:1035] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 243.358 ms
time per row: 0.121679 ms
I20240501 20:09:33.408803 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.133:37155
--local_ip_for_outbound_sockets=127.13.251.133
--tserver_master_addrs=127.13.251.190:44663
--webserver_port=46189
--webserver_interface=127.13.251.133
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 20:09:33.766793 23628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:33.767160 23628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:33.767302 23628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:33.775574 23628 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:33.775746 23628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:33.775899 23628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.133
I20240501 20:09:33.785789 23628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.133:37155
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.13.251.133
--webserver_port=46189
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.133
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:33.787153 23628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:33.789031 23628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:33.801990 23637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:33.802700 23635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:33.803627 23634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:33.804080 23628 server_base.cc:1034] running on GCE node
I20240501 20:09:33.804788 23628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:33.805821 23628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:33.807026 23628 hybrid_clock.cc:648] HybridClock initialized: now 1714594173806965 us; error 55 us; skew 500 ppm
I20240501 20:09:33.807368 23628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:33.809770 23628 webserver.cc:460] Webserver started at http://127.13.251.133:46189/ using document root <none> and password file <none>
I20240501 20:09:33.810542 23628 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:33.810665 23628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:33.817273 23628 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 20:09:33.820286 23643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:33.821776 23628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:33.821943 23628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
format_stamp: "Formatted at 2024-05-01 20:08:59 on dist-test-slave-qswb"
I20240501 20:09:33.822947 23628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:33.847831 23628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:33.848716 23628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:33.849031 23628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:33.850116 23628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:33.851718 23628 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:33.851789 23628 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:33.851881 23628 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:33.851926 23628 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:33.967868 23628 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.133:37155
I20240501 20:09:33.967929 23758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.133:37155 every 8 connection(s)
I20240501 20:09:33.969766 23628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:09:33.975658 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 23628
I20240501 20:09:33.976470 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.134:42431
--local_ip_for_outbound_sockets=127.13.251.134
--tserver_master_addrs=127.13.251.190:44663
--webserver_port=34981
--webserver_interface=127.13.251.134
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:33.987427 23759 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:09:33.987805 23759 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:33.988703 23759 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:09:33.990622 20824 ts_manager.cc:194] Re-registered known tserver with Master: 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:33.991928 20824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.133:38801
W20240501 20:09:34.340655 23763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:34.340948 23763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:34.341079 23763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:34.349225 23763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:34.349426 23763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:34.349648 23763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.134
I20240501 20:09:34.360868 23763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.134:42431
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.13.251.134
--webserver_port=34981
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.134
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:34.362671 23763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:34.364456 23763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:34.376986 23770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:34.377674 23763 server_base.cc:1034] running on GCE node
W20240501 20:09:34.378463 23769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:34.379153 23772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:34.379844 23763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:34.380789 23763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:34.382017 23763 hybrid_clock.cc:648] HybridClock initialized: now 1714594174381946 us; error 65 us; skew 500 ppm
I20240501 20:09:34.382401 23763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:34.384861 23763 webserver.cc:460] Webserver started at http://127.13.251.134:34981/ using document root <none> and password file <none>
I20240501 20:09:34.385636 23763 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:34.385771 23763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:34.391953 23763 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 20:09:34.394836 23778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:34.396479 23763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:34.396646 23763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
format_stamp: "Formatted at 2024-05-01 20:08:59 on dist-test-slave-qswb"
I20240501 20:09:34.397543 23763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:34.424924 23763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:34.425688 23763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:34.425931 23763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:34.426800 23763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:34.428318 23763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:34.428387 23763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:34.428479 23763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:34.428529 23763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:34.472335 23763 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.134:42431
I20240501 20:09:34.472376 23890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.134:42431 every 8 connection(s)
I20240501 20:09:34.473853 23763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:09:34.483278 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 23763
I20240501 20:09:34.483879 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.135:41019
--local_ip_for_outbound_sockets=127.13.251.135
--tserver_master_addrs=127.13.251.190:44663
--webserver_port=39293
--webserver_interface=127.13.251.135
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:34.486660 23891 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:09:34.487004 23891 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:34.487924 23891 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:09:34.489310 20824 ts_manager.cc:194] Re-registered known tserver with Master: c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:34.490227 20824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.134:57325
W20240501 20:09:34.850287 23894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:34.850613 23894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:34.850749 23894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:34.859396 23894 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:34.859563 23894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:34.859737 23894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.135
I20240501 20:09:34.870009 23894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.135:41019
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.13.251.135
--webserver_port=39293
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.135
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:34.871305 23894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:34.873029 23894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:34.886190 23900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:34.886595 23899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:34.886852 23902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:34.887522 23894 server_base.cc:1034] running on GCE node
I20240501 20:09:34.888193 23894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:34.889137 23894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:34.890336 23894 hybrid_clock.cc:648] HybridClock initialized: now 1714594174890285 us; error 38 us; skew 500 ppm
I20240501 20:09:34.890640 23894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:34.892889 23894 webserver.cc:460] Webserver started at http://127.13.251.135:39293/ using document root <none> and password file <none>
I20240501 20:09:34.893630 23894 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:34.893774 23894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:34.900271 23894 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 20:09:34.903071 23908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:34.904620 23894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:34.904744 23894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "744c5a532e724391830c0785d32eb2fb"
format_stamp: "Formatted at 2024-05-01 20:09:00 on dist-test-slave-qswb"
I20240501 20:09:34.905732 23894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:34.942683 23894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:34.943612 23894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:34.943953 23894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:34.944975 23894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:34.946602 23894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:34.946673 23894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:34.946771 23894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:34.946830 23894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:34.994936 23759 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
I20240501 20:09:34.997961 23894 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.135:41019
I20240501 20:09:34.997977 24021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.135:41019 every 8 connection(s)
I20240501 20:09:34.999862 23894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:09:35.000353 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 23894
I20240501 20:09:35.001103 14318 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6p4U5T/build/asan/bin/kudu
/tmp/dist-test-task6p4U5T/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.251.136:44267
--local_ip_for_outbound_sockets=127.13.251.136
--tserver_master_addrs=127.13.251.190:44663
--webserver_port=42049
--webserver_interface=127.13.251.136
--builtin_ntp_servers=127.13.251.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 20:09:35.027143 24022 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:09:35.027472 24022 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:35.028299 24022 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:09:35.029675 20824 ts_manager.cc:194] Re-registered known tserver with Master: 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:35.030447 20824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.135:34035
W20240501 20:09:35.369498 24026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:35.369931 24026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:35.370091 24026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:35.379257 24026 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 20:09:35.379398 24026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:35.379514 24026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.251.136
I20240501 20:09:35.389757 24026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.251.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.251.136:44267
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.13.251.136
--webserver_port=42049
--tserver_master_addrs=127.13.251.190:44663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.251.136
--log_dir=/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:35.391083 24026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:35.393308 24026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:35.406404 24033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:35.407166 24034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:35.407891 24036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:35.408661 24026 server_base.cc:1034] running on GCE node
I20240501 20:09:35.409190 24026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:35.410060 24026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:35.411250 24026 hybrid_clock.cc:648] HybridClock initialized: now 1714594175411207 us; error 34 us; skew 500 ppm
I20240501 20:09:35.411572 24026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:35.413763 24026 webserver.cc:460] Webserver started at http://127.13.251.136:42049/ using document root <none> and password file <none>
I20240501 20:09:35.414563 24026 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:35.414690 24026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:35.420158 24026 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 20:09:35.423161 24042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:35.424887 24026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:35.425048 24026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8559229e044b4e1ea661b881b3f9af50"
format_stamp: "Formatted at 2024-05-01 20:09:00 on dist-test-slave-qswb"
I20240501 20:09:35.426005 24026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:35.468645 24026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:35.469516 24026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:35.469835 24026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:35.470980 24026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:35.472936 24026 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:35.473022 24026 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:35.473155 24026 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:35.473214 24026 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:35.492789 23891 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
I20240501 20:09:35.512800 24026 rpc_server.cc:307] RPC server started. Bound to: 127.13.251.136:44267
I20240501 20:09:35.512838 24154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.251.136:44267 every 8 connection(s)
I20240501 20:09:35.514220 24026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:09:35.517176 14318 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6p4U5T/build/asan/bin/kudu as pid 24026
I20240501 20:09:35.534410 24155 heartbeater.cc:347] Connected to a master server at 127.13.251.190:44663
I20240501 20:09:35.534708 24155 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:35.535728 24155 heartbeater.cc:511] Master 127.13.251.190:44663 requested a full tablet report, sending...
I20240501 20:09:35.535487 20824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:35.537339 20825 ts_manager.cc:194] Re-registered known tserver with Master: 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:35.538261 20825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.251.136:37557
I20240501 20:09:35.562943 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 6be56146ff7749bf8e29844338e401a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=26579ffbfb89410686b7c87ee3113d9f]), partition=
I20240501 20:09:35.563359 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be56146ff7749bf8e29844338e401a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.578698 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 6be56146ff7749bf8e29844338e401a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=26579ffbfb89410686b7c87ee3113d9f]), partition=
I20240501 20:09:35.581209 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be56146ff7749bf8e29844338e401a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.588032 24172 tablet_bootstrap.cc:492] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:35.589779 24172 tablet_bootstrap.cc:654] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.602147 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 6be56146ff7749bf8e29844338e401a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=26579ffbfb89410686b7c87ee3113d9f]), partition=
I20240501 20:09:35.604895 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be56146ff7749bf8e29844338e401a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.606154 24172 tablet_bootstrap.cc:492] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:35.606328 24172 ts_tablet_manager.cc:1367] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20240501 20:09:35.607245 24172 raft_consensus.cc:348] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:35.607466 24172 raft_consensus.cc:374] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.607576 24172 raft_consensus.cc:729] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.607827 24172 consensus_queue.cc:260] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:35.608994 24174 tablet_bootstrap.cc:492] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:35.609385 24172 ts_tablet_manager.cc:1398] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:35.612406 24174 tablet_bootstrap.cc:654] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.613615 24174 log.cc:824] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:35.617141 24174 tablet_bootstrap.cc:492] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:35.617403 24174 ts_tablet_manager.cc:1367] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:09:35.625419 24176 tablet_bootstrap.cc:492] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:35.626658 24174 raft_consensus.cc:348] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:35.626994 24174 raft_consensus.cc:374] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.627146 24174 raft_consensus.cc:729] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.627772 24174 consensus_queue.cc:260] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:35.628926 24022 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
I20240501 20:09:35.628937 24176 tablet_bootstrap.cc:654] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.629479 24174 ts_tablet_manager.cc:1398] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20240501 20:09:35.630261 24176 log.cc:824] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:35.633620 24176 tablet_bootstrap.cc:492] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:35.633966 24176 ts_tablet_manager.cc:1367] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20240501 20:09:35.642524 24176 raft_consensus.cc:348] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:35.642956 24176 raft_consensus.cc:374] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.643061 24176 raft_consensus.cc:729] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.643909 24176 consensus_queue.cc:260] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:35.645695 24176 ts_tablet_manager.cc:1398] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
W20240501 20:09:35.722254 21013 tablet.cc:2369] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:09:35.722563 23760 tablet.cc:2369] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:09:35.763582 24023 tablet.cc:2369] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:35.794643 23573 raft_consensus.cc:482] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:35.794832 23573 raft_consensus.cc:504] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:35.797062 23573 leader_election.cc:288] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:35.804209 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be56146ff7749bf8e29844338e401a5" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:35.804694 23712 raft_consensus.cc:2454] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:35.805205 20902 leader_election.cc:302] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:35.805837 23573 raft_consensus.cc:2781] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:35.805972 23573 raft_consensus.cc:482] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:35.806061 23573 raft_consensus.cc:3037] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.808837 23573 raft_consensus.cc:504] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:35.810012 23573 leader_election.cc:288] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:35.810982 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be56146ff7749bf8e29844338e401a5" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:35.811082 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be56146ff7749bf8e29844338e401a5" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:35.811337 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be56146ff7749bf8e29844338e401a5" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:35.811522 23976 raft_consensus.cc:3037] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.811553 23712 raft_consensus.cc:3037] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.814646 23712 raft_consensus.cc:2454] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:35.815018 20902 leader_election.cc:302] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:35.815371 23573 raft_consensus.cc:2781] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:35.815433 23976 raft_consensus.cc:2454] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:35.815580 23573 raft_consensus.cc:686] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:35.816001 23573 consensus_queue.cc:237] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:35.820555 20824 catalog_manager.cc:5484] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:35.951954 20824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:35.961020 23976 raft_consensus.cc:1261] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:35.961915 23573 consensus_queue.cc:1035] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:35.962651 23712 raft_consensus.cc:1261] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:35.963920 23573 consensus_queue.cc:1035] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:35.972424 20825 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet 6be56146ff7749bf8e29844338e401a5 (table rebalancer_dynamic_table [id=26579ffbfb89410686b7c87ee3113d9f]). Expected version 1 got 0
I20240501 20:09:35.979897 24187 mvcc.cc:205] Tried to move back new op lower bound from 7022977744726929408 to 7022977744144846848. Current Snapshot: MvccSnapshot[applied={T|T < 7022977744726929408}]
I20240501 20:09:35.983953 24193 tablet.cc:1722] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:35.986187 24193 tablet.cc:1620] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:35.998172 20824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=26579ffbfb89410686b7c87ee3113d9f] alter complete (version 1)
I20240501 20:09:36.017845 24194 tablet.cc:1722] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:36.019717 24194 tablet.cc:1620] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:36.019717 20824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:36.020213 20825 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet 6be56146ff7749bf8e29844338e401a5 (table rebalancer_dynamic_table [id=26579ffbfb89410686b7c87ee3113d9f]). Expected version 1 got 0
I20240501 20:09:36.030030 20824 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 20:09:36.030300 20824 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 20:09:36.038195 20824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 398c2a957b7547ad9a92729242f5792f
I20240501 20:09:36.039522 20824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 41558083a68b4ab09f4e762188537761
I20240501 20:09:36.040933 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 41558083a68b4ab09f4e762188537761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:36.041276 23957 tablet_service.cc:1467] Processing CreateTablet for tablet e9cc62237cde4a90b386f5c30625445c (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.041993 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9cc62237cde4a90b386f5c30625445c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.046200 20824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 67e11df69820456d8aa2ba056ca29a5d
I20240501 20:09:36.046219 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 939b60669869461483f50e61d8dc633b (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.046936 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939b60669869461483f50e61d8dc633b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.047189 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e11df69820456d8aa2ba056ca29a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:36.050041 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 398c2a957b7547ad9a92729242f5792f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:36.051066 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 398c2a957b7547ad9a92729242f5792f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:36.057514 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet 41558083a68b4ab09f4e762188537761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:36.057607 20946 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e11df69820456d8aa2ba056ca29a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:36.060247 21337 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e11df69820456d8aa2ba056ca29a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:36.071559 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 41558083a68b4ab09f4e762188537761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:36.072098 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 7a23f60c87794a49ae1f82dff06fe881 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.072646 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a23f60c87794a49ae1f82dff06fe881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.071419 21078 tablet_service.cc:1514] Processing DeleteTablet for tablet 398c2a957b7547ad9a92729242f5792f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:36.088680 24176 tablet_bootstrap.cc:492] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:36.091121 24176 tablet_bootstrap.cc:654] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.094384 24174 tablet_bootstrap.cc:492] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:36.096526 24174 tablet_bootstrap.cc:654] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.102739 24176 tablet_bootstrap.cc:492] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:36.103000 24176 ts_tablet_manager.cc:1367] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240501 20:09:36.103197 24203 tablet_replica.cc:330] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:36.106169 24203 raft_consensus.cc:2229] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.109443 24203 raft_consensus.cc:2258] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.111721 24203 ts_tablet_manager.cc:1875] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.111812 20824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:36.116940 24176 raft_consensus.cc:348] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.117367 24176 raft_consensus.cc:374] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.117527 24176 raft_consensus.cc:729] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.117884 24176 consensus_queue.cc:260] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.118745 24176 ts_tablet_manager.cc:1398] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:09:36.121891 24201 tablet_replica.cc:330] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:36.123339 24204 tablet_bootstrap.cc:492] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:36.123420 24201 raft_consensus.cc:2229] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.124219 24174 tablet_bootstrap.cc:492] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:36.126140 24204 tablet_bootstrap.cc:654] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.126354 24174 ts_tablet_manager.cc:1367] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.006s
I20240501 20:09:36.127461 24174 raft_consensus.cc:348] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.128585 24174 raft_consensus.cc:374] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.128729 24174 raft_consensus.cc:729] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.129448 24201 raft_consensus.cc:2258] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.129086 24174 consensus_queue.cc:260] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.136693 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 939b60669869461483f50e61d8dc633b (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.137836 24174 ts_tablet_manager.cc:1398] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 20:09:36.138063 24201 ts_tablet_manager.cc:1875] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.138135 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939b60669869461483f50e61d8dc633b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.140326 24193 tablet.cc:1722] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:36.140859 24193 tablet.cc:1620] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:36.140931 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 7a23f60c87794a49ae1f82dff06fe881 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.141474 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a23f60c87794a49ae1f82dff06fe881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.142212 24194 tablet.cc:1722] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:36.142661 24194 tablet.cc:1620] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:36.135179 24090 tablet_service.cc:1467] Processing CreateTablet for tablet e9cc62237cde4a90b386f5c30625445c (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.147450 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9cc62237cde4a90b386f5c30625445c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.151022 20825 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=26579ffbfb89410686b7c87ee3113d9f] alter complete (version 2)
I20240501 20:09:36.152665 24197 tablet_replica.cc:330] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:36.153970 24197 raft_consensus.cc:2229] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.154358 24197 raft_consensus.cc:2258] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 20:09:36.155310 21032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 154795us
I20240501 20:09:36.156256 24204 tablet_bootstrap.cc:492] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:36.156570 24204 ts_tablet_manager.cc:1367] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.000s
I20240501 20:09:36.157963 24204 raft_consensus.cc:348] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.158499 24204 raft_consensus.cc:374] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.158782 24204 raft_consensus.cc:729] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.159284 24204 consensus_queue.cc:260] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.159610 24203 ts_tablet_manager.cc:1888] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.160025 24204 ts_tablet_manager.cc:1398] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:36.160897 24203 log.cc:1196] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/41558083a68b4ab09f4e762188537761
I20240501 20:09:36.162683 24200 tablet_replica.cc:330] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:36.163323 24200 raft_consensus.cc:2229] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:36.164412 24200 raft_consensus.cc:2258] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.167291 24203 ts_tablet_manager.cc:1909] T 41558083a68b4ab09f4e762188537761 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:36.168833 24203 tablet_replica.cc:330] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:36.169005 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 41558083a68b4ab09f4e762188537761 (table rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]) successfully deleted
I20240501 20:09:36.169092 24203 raft_consensus.cc:2229] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.169306 24203 raft_consensus.cc:2258] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.169696 24197 ts_tablet_manager.cc:1875] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.169903 24203 ts_tablet_manager.cc:1875] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.172394 24200 ts_tablet_manager.cc:1875] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.175148 24203 ts_tablet_manager.cc:1888] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.175284 24203 log.cc:1196] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/398c2a957b7547ad9a92729242f5792f
I20240501 20:09:36.175657 24203 ts_tablet_manager.cc:1909] T 398c2a957b7547ad9a92729242f5792f P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:36.176622 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 398c2a957b7547ad9a92729242f5792f (table rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]) successfully deleted
I20240501 20:09:36.177484 24209 tablet_bootstrap.cc:492] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:36.178212 24201 ts_tablet_manager.cc:1888] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.178472 24201 log.cc:1196] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/398c2a957b7547ad9a92729242f5792f
I20240501 20:09:36.179764 24201 ts_tablet_manager.cc:1909] T 398c2a957b7547ad9a92729242f5792f P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
W20240501 20:09:36.180225 21144 tablet.cc:2369] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:36.182206 24209 tablet_bootstrap.cc:654] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.186167 24201 tablet_replica.cc:330] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:36.186594 24201 raft_consensus.cc:2229] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:36.186882 20825 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 6be56146ff7749bf8e29844338e401a5 (table rebalancer_dynamic_table [id=26579ffbfb89410686b7c87ee3113d9f]). Expected version 2 got 0
I20240501 20:09:36.186359 23826 tablet_service.cc:1467] Processing CreateTablet for tablet e9cc62237cde4a90b386f5c30625445c (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.187645 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 398c2a957b7547ad9a92729242f5792f (table rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]) successfully deleted
I20240501 20:09:36.187409 24201 raft_consensus.cc:2258] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.189046 24201 ts_tablet_manager.cc:1875] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.193621 24190 mvcc.cc:205] Tried to move back new op lower bound from 7022977744726929408 to 7022977744144846848. Current Snapshot: MvccSnapshot[applied={T|T < 7022977744726929408}]
I20240501 20:09:36.188073 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 939b60669869461483f50e61d8dc633b (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.198865 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939b60669869461483f50e61d8dc633b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.199776 24201 ts_tablet_manager.cc:1888] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.199950 24201 log.cc:1196] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/41558083a68b4ab09f4e762188537761
I20240501 20:09:36.200429 24201 ts_tablet_manager.cc:1909] T 41558083a68b4ab09f4e762188537761 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:36.201653 24201 tablet_replica.cc:330] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:36.202344 24201 raft_consensus.cc:2229] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:36.202813 24201 raft_consensus.cc:2258] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.203843 24201 ts_tablet_manager.cc:1875] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.192135 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 7a23f60c87794a49ae1f82dff06fe881 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.204277 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 41558083a68b4ab09f4e762188537761 (table rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]) successfully deleted
I20240501 20:09:36.204555 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a23f60c87794a49ae1f82dff06fe881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.205431 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9cc62237cde4a90b386f5c30625445c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.205624 24195 raft_consensus.cc:482] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.205809 24195 raft_consensus.cc:504] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.212812 24201 ts_tablet_manager.cc:1888] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.212971 24201 log.cc:1196] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/67e11df69820456d8aa2ba056ca29a5d
I20240501 20:09:36.213521 24201 ts_tablet_manager.cc:1909] T 67e11df69820456d8aa2ba056ca29a5d P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:36.214776 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 67e11df69820456d8aa2ba056ca29a5d (table rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]) successfully deleted
I20240501 20:09:36.216882 24195 leader_election.cc:288] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:36.235834 24200 ts_tablet_manager.cc:1888] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.237406 24197 ts_tablet_manager.cc:1888] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.237735 24200 log.cc:1196] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/398c2a957b7547ad9a92729242f5792f
I20240501 20:09:36.238575 24200 ts_tablet_manager.cc:1909] T 398c2a957b7547ad9a92729242f5792f P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:36.239624 24197 log.cc:1196] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/41558083a68b4ab09f4e762188537761
I20240501 20:09:36.240408 24197 ts_tablet_manager.cc:1909] T 41558083a68b4ab09f4e762188537761 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:36.242664 24200 tablet_replica.cc:330] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:36.242724 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 398c2a957b7547ad9a92729242f5792f (table rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]) successfully deleted
I20240501 20:09:36.244901 24197 tablet_replica.cc:330] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:36.245007 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 41558083a68b4ab09f4e762188537761 (table rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]) successfully deleted
I20240501 20:09:36.245393 24200 raft_consensus.cc:2229] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.245736 24200 raft_consensus.cc:2258] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.245925 24209 log.cc:824] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:36.247014 24200 ts_tablet_manager.cc:1875] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.247208 24197 raft_consensus.cc:2229] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.247608 24197 raft_consensus.cc:2258] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.248675 24197 ts_tablet_manager.cc:1875] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.254863 20825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:36.263794 24193 tablet.cc:1722] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:36.264885 24197 ts_tablet_manager.cc:1888] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.265244 24194 tablet.cc:1722] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:36.265911 24200 ts_tablet_manager.cc:1888] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.266266 24197 log.cc:1196] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/67e11df69820456d8aa2ba056ca29a5d
I20240501 20:09:36.266773 24197 ts_tablet_manager.cc:1909] T 67e11df69820456d8aa2ba056ca29a5d P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:36.266944 24200 log.cc:1196] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/67e11df69820456d8aa2ba056ca29a5d
I20240501 20:09:36.266898 24210 tablet.cc:1722] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:36.267519 24200 ts_tablet_manager.cc:1909] T 67e11df69820456d8aa2ba056ca29a5d P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:36.268721 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 67e11df69820456d8aa2ba056ca29a5d (table rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]) successfully deleted
I20240501 20:09:36.270490 20821 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 6be56146ff7749bf8e29844338e401a5 (table rebalancer_dynamic_table_new_name [id=26579ffbfb89410686b7c87ee3113d9f]). Expected version 3 got 0
I20240501 20:09:36.272562 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 67e11df69820456d8aa2ba056ca29a5d (table rebalance_test_table_0 [id=e43b06cd741549fdac33c962a24d7907]) successfully deleted
I20240501 20:09:36.277513 20825 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=26579ffbfb89410686b7c87ee3113d9f] alter complete (version 3)
I20240501 20:09:36.282675 24209 tablet_bootstrap.cc:492] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:36.284476 24210 tablet.cc:1620] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:36.286969 24209 ts_tablet_manager.cc:1367] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.110s user 0.000s sys 0.009s
I20240501 20:09:36.288652 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a23f60c87794a49ae1f82dff06fe881" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:09:36.296485 21034 leader_election.cc:341] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:36.302680 24215 tablet_bootstrap.cc:492] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:36.304792 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a23f60c87794a49ae1f82dff06fe881" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
W20240501 20:09:36.306210 21033 leader_election.cc:341] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:36.306366 21033 leader_election.cc:302] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feac73db0a3b4aafa374ca5dfd322b08; no voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:36.306778 24195 raft_consensus.cc:2726] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:36.307150 20825 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 6be56146ff7749bf8e29844338e401a5 (table rebalancer_dynamic_table_new_name [id=26579ffbfb89410686b7c87ee3113d9f]). Expected version 3 got 1
I20240501 20:09:36.309787 24209 raft_consensus.cc:348] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.310240 24209 raft_consensus.cc:374] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.310349 24209 raft_consensus.cc:729] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.310892 24210 tablet.cc:1722] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:36.311586 24215 tablet_bootstrap.cc:654] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.311630 24209 consensus_queue.cc:260] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.311836 24210 tablet.cc:1620] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:36.315465 24215 log.cc:824] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:36.320133 20825 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 6be56146ff7749bf8e29844338e401a5 (table rebalancer_dynamic_table_new_name [id=26579ffbfb89410686b7c87ee3113d9f]). Expected version 3 got 2
I20240501 20:09:36.324075 24215 tablet_bootstrap.cc:492] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:36.324549 24215 ts_tablet_manager.cc:1367] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20240501 20:09:36.328475 24210 tablet.cc:1722] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:36.329427 24155 heartbeater.cc:503] Master 127.13.251.190:44663 was elected leader, sending a full tablet report...
I20240501 20:09:36.330559 24209 ts_tablet_manager.cc:1398] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.043s user 0.021s sys 0.008s
I20240501 20:09:36.332314 24209 tablet_bootstrap.cc:492] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:36.335268 24177 raft_consensus.cc:482] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.335454 24209 tablet_bootstrap.cc:654] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.335450 24177 raft_consensus.cc:504] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.337344 24177 leader_election.cc:288] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:36.341658 24215 raft_consensus.cc:348] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.342083 24215 raft_consensus.cc:374] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.342325 24215 raft_consensus.cc:729] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.344246 24215 consensus_queue.cc:260] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.350587 24209 tablet_bootstrap.cc:492] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:36.350749 24209 ts_tablet_manager.cc:1367] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20240501 20:09:36.351883 24209 raft_consensus.cc:348] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.352134 24209 raft_consensus.cc:374] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.352229 24209 raft_consensus.cc:729] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.352581 24209 consensus_queue.cc:260] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.353308 24209 ts_tablet_manager.cc:1398] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:36.353646 24209 tablet_bootstrap.cc:492] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:36.357187 24215 ts_tablet_manager.cc:1398] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.032s user 0.008s sys 0.004s
I20240501 20:09:36.357746 24215 tablet_bootstrap.cc:492] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:36.359966 24215 tablet_bootstrap.cc:654] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.363129 24179 raft_consensus.cc:482] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.363315 24179 raft_consensus.cc:504] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.365047 24209 tablet_bootstrap.cc:654] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.365510 24179 leader_election.cc:288] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:36.366120 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9cc62237cde4a90b386f5c30625445c" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
W20240501 20:09:36.367830 23911 leader_election.cc:341] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:36.376798 24215 tablet_bootstrap.cc:492] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:36.377086 24215 ts_tablet_manager.cc:1367] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20240501 20:09:36.376999 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9cc62237cde4a90b386f5c30625445c" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:09:36.377866 23912 leader_election.cc:341] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:36.378013 23912 leader_election.cc:302] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb; no voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:36.378800 24177 raft_consensus.cc:2726] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:36.378801 24215 raft_consensus.cc:348] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.379009 24215 raft_consensus.cc:374] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.379191 24215 raft_consensus.cc:729] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.379473 24215 consensus_queue.cc:260] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.379990 24209 tablet_bootstrap.cc:492] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:36.380043 24215 ts_tablet_manager.cc:1398] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:36.380272 24209 ts_tablet_manager.cc:1367] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.006s
I20240501 20:09:36.380357 24215 tablet_bootstrap.cc:492] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:36.381140 24209 raft_consensus.cc:348] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.381505 24209 raft_consensus.cc:374] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.381635 24209 raft_consensus.cc:729] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.382781 24209 consensus_queue.cc:260] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.384155 24209 ts_tablet_manager.cc:1398] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:09:36.384379 24215 tablet_bootstrap.cc:654] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.386152 24224 raft_consensus.cc:482] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.386425 24224 raft_consensus.cc:504] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.388513 24224 leader_election.cc:288] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:36.389503 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939b60669869461483f50e61d8dc633b" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:36.389844 24110 raft_consensus.cc:2454] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:36.390133 20824 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:36.390347 20824 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:36.390429 23646 leader_election.cc:302] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:36.390913 24179 raft_consensus.cc:2781] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.391036 24179 raft_consensus.cc:482] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.391151 24179 raft_consensus.cc:3037] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.393915 24179 raft_consensus.cc:504] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.394668 24179 leader_election.cc:288] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:36.395490 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939b60669869461483f50e61d8dc633b" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:36.395735 24110 raft_consensus.cc:3037] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.398870 24110 raft_consensus.cc:2454] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:36.399179 20824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 6be56146ff7749bf8e29844338e401a5
I20240501 20:09:36.399309 23646 leader_election.cc:302] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:36.400588 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be56146ff7749bf8e29844338e401a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:36.400694 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be56146ff7749bf8e29844338e401a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:36.400991 24201 tablet_replica.cc:330] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:36.401315 24201 raft_consensus.cc:2229] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:36.401691 24201 raft_consensus.cc:2258] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.401788 24179 raft_consensus.cc:2781] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.402520 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be56146ff7749bf8e29844338e401a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:36.405491 24201 ts_tablet_manager.cc:1875] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.418373 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939b60669869461483f50e61d8dc633b" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:36.418721 23846 raft_consensus.cc:2454] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:36.419174 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939b60669869461483f50e61d8dc633b" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:36.419502 23845 raft_consensus.cc:3037] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.432564 23845 raft_consensus.cc:2454] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:36.435288 24179 raft_consensus.cc:686] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:36.435851 24179 consensus_queue.cc:237] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:36.442047 24201 ts_tablet_manager.cc:1888] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:36.456669 24201 log.cc:1196] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/6be56146ff7749bf8e29844338e401a5
I20240501 20:09:36.457211 24201 ts_tablet_manager.cc:1909] T 6be56146ff7749bf8e29844338e401a5 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:36.458376 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 6be56146ff7749bf8e29844338e401a5 (table rebalancer_dynamic_table_new_name [id=26579ffbfb89410686b7c87ee3113d9f]) successfully deleted
I20240501 20:09:36.461936 20824 catalog_manager.cc:5484] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:36.475097 24237 tablet_replica.cc:330] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
W20240501 20:09:36.479571 23892 tablet.cc:2369] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:36.480700 24237 raft_consensus.cc:2229] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.481122 24237 raft_consensus.cc:2258] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.483285 24235 tablet_replica.cc:330] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:36.483987 24235 raft_consensus.cc:2229] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.485952 24237 ts_tablet_manager.cc:1875] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.495222 24237 ts_tablet_manager.cc:1888] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:36.495272 24235 raft_consensus.cc:2258] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.502077 24215 tablet_bootstrap.cc:492] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:36.502305 24215 ts_tablet_manager.cc:1367] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.122s user 0.012s sys 0.018s
I20240501 20:09:36.503386 24215 raft_consensus.cc:348] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.503676 24215 raft_consensus.cc:374] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.503767 24215 raft_consensus.cc:729] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.504072 24215 consensus_queue.cc:260] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.504616 24215 ts_tablet_manager.cc:1398] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:36.495520 24237 log.cc:1196] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/6be56146ff7749bf8e29844338e401a5
I20240501 20:09:36.505261 24237 ts_tablet_manager.cc:1909] T 6be56146ff7749bf8e29844338e401a5 P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:36.506808 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 6be56146ff7749bf8e29844338e401a5 (table rebalancer_dynamic_table_new_name [id=26579ffbfb89410686b7c87ee3113d9f]) successfully deleted
I20240501 20:09:36.508450 24235 ts_tablet_manager.cc:1875] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.509059 24224 raft_consensus.cc:482] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.509222 24224 raft_consensus.cc:504] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.509860 24224 leader_election.cc:288] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
W20240501 20:09:36.516404 24156 tablet.cc:2369] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:36.516276 20824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:36.529424 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 58420a59ee814bf8ad558f693bd460f3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=430664710eff4ee485a54c3ca3fbfffd]), partition=
I20240501 20:09:36.529559 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939b60669869461483f50e61d8dc633b" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:36.529841 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58420a59ee814bf8ad558f693bd460f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.529978 24110 raft_consensus.cc:2379] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in this term.
I20240501 20:09:36.531348 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939b60669869461483f50e61d8dc633b" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:36.532179 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 58420a59ee814bf8ad558f693bd460f3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=430664710eff4ee485a54c3ca3fbfffd]), partition=
I20240501 20:09:36.532171 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a23f60c87794a49ae1f82dff06fe881" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:36.532371 24110 raft_consensus.cc:2454] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:36.532553 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58420a59ee814bf8ad558f693bd460f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.532774 23781 leader_election.cc:302] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:36.532939 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 58420a59ee814bf8ad558f693bd460f3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=430664710eff4ee485a54c3ca3fbfffd]), partition=
I20240501 20:09:36.533211 24224 raft_consensus.cc:2781] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.533298 24224 raft_consensus.cc:482] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.533320 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58420a59ee814bf8ad558f693bd460f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.533383 24224 raft_consensus.cc:3037] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.536229 24224 raft_consensus.cc:504] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.536916 24224 leader_election.cc:288] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:36.537297 23781 leader_election.cc:302] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:36.538201 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a23f60c87794a49ae1f82dff06fe881" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:36.538403 24110 raft_consensus.cc:3037] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.541155 24110 raft_consensus.cc:2454] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:36.541280 24176 tablet_bootstrap.cc:492] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:36.541533 24224 raft_consensus.cc:2726] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:36.541908 23781 leader_election.cc:302] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:36.542273 24224 raft_consensus.cc:2781] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.542923 24176 tablet_bootstrap.cc:654] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.558661 24176 tablet_bootstrap.cc:492] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:36.558861 24176 ts_tablet_manager.cc:1367] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240501 20:09:36.559733 24176 raft_consensus.cc:348] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:36.560002 24176 raft_consensus.cc:374] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.560084 24176 raft_consensus.cc:729] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.560370 24176 consensus_queue.cc:260] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:36.564113 24176 ts_tablet_manager.cc:1398] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:36.570667 24244 tablet_bootstrap.cc:492] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:36.571326 24245 tablet_bootstrap.cc:492] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:36.578944 24245 tablet_bootstrap.cc:654] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.579341 20824 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 20:09:36.579655 20824 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 20:09:36.580041 24244 tablet_bootstrap.cc:654] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.587124 20824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 80890a0703c2467fbdc2caaeec72ea7d
I20240501 20:09:36.587894 20824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 15e23b998afc4b0da79c87476bd55af6
I20240501 20:09:36.588286 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 80890a0703c2467fbdc2caaeec72ea7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:36.588408 20824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet f4cb707f4780467dbe97fc03978004fe
I20240501 20:09:36.588735 24200 tablet_replica.cc:330] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:36.588984 24200 raft_consensus.cc:2229] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.589251 24200 raft_consensus.cc:2258] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.589684 24245 tablet_bootstrap.cc:492] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:36.589825 24245 ts_tablet_manager.cc:1367] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240501 20:09:36.589967 24200 ts_tablet_manager.cc:1875] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.590831 24235 ts_tablet_manager.cc:1888] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:36.590994 24244 tablet_bootstrap.cc:492] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:36.591044 24235 log.cc:1196] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/6be56146ff7749bf8e29844338e401a5
I20240501 20:09:36.591132 24244 ts_tablet_manager.cc:1367] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240501 20:09:36.591676 21337 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cb707f4780467dbe97fc03978004fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:36.591954 24244 raft_consensus.cc:348] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:36.592443 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 15e23b998afc4b0da79c87476bd55af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:36.593372 24245 raft_consensus.cc:348] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:36.594986 24235 ts_tablet_manager.cc:1909] T 6be56146ff7749bf8e29844338e401a5 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:36.595575 24244 raft_consensus.cc:374] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.595853 24245 raft_consensus.cc:374] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.596037 24244 raft_consensus.cc:729] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.596287 24245 raft_consensus.cc:729] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.596616 24244 consensus_queue.cc:260] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:36.594497 20946 tablet_service.cc:1514] Processing DeleteTablet for tablet 15e23b998afc4b0da79c87476bd55af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:36.597223 24245 consensus_queue.cc:260] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:36.597478 24201 tablet_replica.cc:330] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:36.594558 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cb707f4780467dbe97fc03978004fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:36.597808 24245 ts_tablet_manager.cc:1398] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:09:36.597949 24201 raft_consensus.cc:2229] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:36.598306 24201 raft_consensus.cc:2258] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.599061 24201 ts_tablet_manager.cc:1875] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.599638 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet 80890a0703c2467fbdc2caaeec72ea7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:36.600064 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cb707f4780467dbe97fc03978004fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:36.600251 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 6be56146ff7749bf8e29844338e401a5 (table rebalancer_dynamic_table_new_name [id=26579ffbfb89410686b7c87ee3113d9f]) successfully deleted
I20240501 20:09:36.600642 24203 tablet_replica.cc:330] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:36.600904 24203 raft_consensus.cc:2229] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.601248 24203 raft_consensus.cc:2258] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.601552 24244 ts_tablet_manager.cc:1398] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:09:36.602077 24203 ts_tablet_manager.cc:1875] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.605104 24179 raft_consensus.cc:482] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.605288 24179 raft_consensus.cc:504] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:36.606091 24179 leader_election.cc:288] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:36.613720 24203 ts_tablet_manager.cc:1888] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.613885 24203 log.cc:1196] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/80890a0703c2467fbdc2caaeec72ea7d
I20240501 20:09:36.614459 24203 ts_tablet_manager.cc:1909] T 80890a0703c2467fbdc2caaeec72ea7d P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:36.618249 24224 raft_consensus.cc:686] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:36.619031 24224 consensus_queue.cc:237] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:36.622462 24203 tablet_replica.cc:330] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:36.622793 24203 raft_consensus.cc:2229] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:36.625330 24203 raft_consensus.cc:2258] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.625396 24201 ts_tablet_manager.cc:1888] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.625579 24201 log.cc:1196] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/15e23b998afc4b0da79c87476bd55af6
I20240501 20:09:36.626029 24201 ts_tablet_manager.cc:1909] T 15e23b998afc4b0da79c87476bd55af6 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
W20240501 20:09:36.626284 20900 consensus_peers.cc:485] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80890a0703c2467fbdc2caaeec72ea7d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:36.629351 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 80890a0703c2467fbdc2caaeec72ea7d (table rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]) successfully deleted
I20240501 20:09:36.631039 24200 ts_tablet_manager.cc:1888] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.631896 24203 ts_tablet_manager.cc:1875] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.634420 24200 log.cc:1196] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/80890a0703c2467fbdc2caaeec72ea7d
I20240501 20:09:36.634982 24200 ts_tablet_manager.cc:1909] T 80890a0703c2467fbdc2caaeec72ea7d P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
W20240501 20:09:36.635262 21274 tablet.cc:2369] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:36.638239 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 80890a0703c2467fbdc2caaeec72ea7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:36.636655 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 15e23b998afc4b0da79c87476bd55af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:36 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:36.640961 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 80890a0703c2467fbdc2caaeec72ea7d (table rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]) successfully deleted
I20240501 20:09:36.641378 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a23f60c87794a49ae1f82dff06fe881" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:36.641477 24197 tablet_replica.cc:330] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:36.641605 21097 raft_consensus.cc:2454] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:36.641707 24197 raft_consensus.cc:2229] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.641993 24197 raft_consensus.cc:2258] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.642143 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a23f60c87794a49ae1f82dff06fe881" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:36.642226 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 15e23b998afc4b0da79c87476bd55af6 (table rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]) successfully deleted
I20240501 20:09:36.642339 21097 raft_consensus.cc:3037] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.642544 24200 tablet_replica.cc:330] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:36.642766 24197 ts_tablet_manager.cc:1875] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.642769 24200 raft_consensus.cc:2229] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.643008 24200 raft_consensus.cc:2258] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.643721 24200 ts_tablet_manager.cc:1875] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.642078 24201 tablet_replica.cc:330] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:36.644338 24201 raft_consensus.cc:2229] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.645098 21097 raft_consensus.cc:2454] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:36.646183 24201 raft_consensus.cc:2258] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.647933 24201 ts_tablet_manager.cc:1875] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.650107 24197 ts_tablet_manager.cc:1888] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.650378 24197 log.cc:1196] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/15e23b998afc4b0da79c87476bd55af6
I20240501 20:09:36.650967 24197 ts_tablet_manager.cc:1909] T 15e23b998afc4b0da79c87476bd55af6 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:36.651010 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58420a59ee814bf8ad558f693bd460f3" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:36.651294 21226 raft_consensus.cc:2454] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:36.651876 23647 leader_election.cc:302] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46; no voters:
I20240501 20:09:36.652271 24179 raft_consensus.cc:2781] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.652365 24179 raft_consensus.cc:482] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.652459 24179 raft_consensus.cc:3037] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.652541 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 15e23b998afc4b0da79c87476bd55af6 (table rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]) successfully deleted
I20240501 20:09:36.652304 20824 catalog_manager.cc:5484] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:36.654954 24179 raft_consensus.cc:504] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:36.656656 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58420a59ee814bf8ad558f693bd460f3" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:36.656883 21226 raft_consensus.cc:3037] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.656958 24203 ts_tablet_manager.cc:1888] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.657162 24203 log.cc:1196] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/f4cb707f4780467dbe97fc03978004fe
I20240501 20:09:36.657696 24203 ts_tablet_manager.cc:1909] T f4cb707f4780467dbe97fc03978004fe P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:36.659075 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet f4cb707f4780467dbe97fc03978004fe (table rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]) successfully deleted
I20240501 20:09:36.660559 24179 leader_election.cc:288] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:36.660897 21226 raft_consensus.cc:2454] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:36.661952 23647 leader_election.cc:302] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46; no voters:
I20240501 20:09:36.662345 24179 raft_consensus.cc:2781] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.664857 24179 raft_consensus.cc:686] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:36.665208 24179 consensus_queue.cc:237] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:36.668331 20824 catalog_manager.cc:5484] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:36.669036 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58420a59ee814bf8ad558f693bd460f3" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:36.669273 20966 raft_consensus.cc:2454] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:36.669080 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58420a59ee814bf8ad558f693bd460f3" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:36.669647 20965 raft_consensus.cc:3037] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.670018 24201 ts_tablet_manager.cc:1888] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.670223 24201 log.cc:1196] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/f4cb707f4780467dbe97fc03978004fe
I20240501 20:09:36.670862 24201 ts_tablet_manager.cc:1909] T f4cb707f4780467dbe97fc03978004fe P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:36.672524 20965 raft_consensus.cc:2454] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:36.674597 24200 ts_tablet_manager.cc:1888] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.674798 24200 log.cc:1196] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/f4cb707f4780467dbe97fc03978004fe
I20240501 20:09:36.675316 24200 ts_tablet_manager.cc:1909] T f4cb707f4780467dbe97fc03978004fe P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:36.676704 24201 tablet_replica.cc:330] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:36.676795 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet f4cb707f4780467dbe97fc03978004fe (table rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]) successfully deleted
I20240501 20:09:36.677695 24200 tablet_replica.cc:330] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:36.677771 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet f4cb707f4780467dbe97fc03978004fe (table rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]) successfully deleted
I20240501 20:09:36.676992 24201 raft_consensus.cc:2229] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:36.678364 24201 raft_consensus.cc:2258] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.679316 24201 ts_tablet_manager.cc:1875] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.687768 24200 raft_consensus.cc:2229] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:36.688063 24200 raft_consensus.cc:2258] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:36.688761 24200 ts_tablet_manager.cc:1875] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:36.691776 24201 ts_tablet_manager.cc:1888] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.691915 24201 log.cc:1196] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/80890a0703c2467fbdc2caaeec72ea7d
I20240501 20:09:36.692361 24201 ts_tablet_manager.cc:1909] T 80890a0703c2467fbdc2caaeec72ea7d P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:36.693578 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 80890a0703c2467fbdc2caaeec72ea7d (table rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]) successfully deleted
I20240501 20:09:36.695027 24200 ts_tablet_manager.cc:1888] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:36.695142 24200 log.cc:1196] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/15e23b998afc4b0da79c87476bd55af6
I20240501 20:09:36.695520 24200 ts_tablet_manager.cc:1909] T 15e23b998afc4b0da79c87476bd55af6 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:36.696524 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 15e23b998afc4b0da79c87476bd55af6 (table rebalance_test_table_2 [id=4a85e0308f104c05b0f6dca6f66a2f97]) successfully deleted
I20240501 20:09:36.715757 24217 raft_consensus.cc:482] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.715973 24217 raft_consensus.cc:504] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.717849 24217 leader_election.cc:288] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:36.735321 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9cc62237cde4a90b386f5c30625445c" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:36.735667 23845 raft_consensus.cc:2454] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:36.738282 24046 leader_election.cc:302] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:36.738842 24217 raft_consensus.cc:2781] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.738979 24217 raft_consensus.cc:482] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.739082 24217 raft_consensus.cc:3037] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.742095 24217 raft_consensus.cc:504] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.747365 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9cc62237cde4a90b386f5c30625445c" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:36.747673 23845 raft_consensus.cc:3037] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.750751 23845 raft_consensus.cc:2454] T e9cc62237cde4a90b386f5c30625445c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:36.750696 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9cc62237cde4a90b386f5c30625445c" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:36.750932 23976 raft_consensus.cc:2454] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:36.751250 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9cc62237cde4a90b386f5c30625445c" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:36.751444 23977 raft_consensus.cc:3037] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.752362 24046 leader_election.cc:302] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:36.752851 24217 leader_election.cc:288] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:36.753108 24217 raft_consensus.cc:2781] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.753286 24217 raft_consensus.cc:686] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:36.753836 24217 consensus_queue.cc:237] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:36.758213 23977 raft_consensus.cc:2454] T e9cc62237cde4a90b386f5c30625445c P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:36.767449 20824 catalog_manager.cc:5484] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:36.787910 20824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:36.797531 20965 raft_consensus.cc:1261] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:36.797559 21226 raft_consensus.cc:1261] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:36.798164 24236 consensus_queue.cc:1035] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:36.798852 24236 consensus_queue.cc:1035] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:36.832139 20824 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet 58420a59ee814bf8ad558f693bd460f3 (table rebalancer_dynamic_table [id=430664710eff4ee485a54c3ca3fbfffd]). Expected version 1 got 0
I20240501 20:09:36.835280 24220 tablet.cc:1722] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:36.837306 24220 tablet.cc:1620] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:36.848794 20824 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 58420a59ee814bf8ad558f693bd460f3 (table rebalancer_dynamic_table [id=430664710eff4ee485a54c3ca3fbfffd]). Expected version 1 got 0
I20240501 20:09:36.849943 24287 tablet.cc:1722] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:36.859285 24287 tablet.cc:1620] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:36.859771 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=430664710eff4ee485a54c3ca3fbfffd] alter complete (version 1)
I20240501 20:09:36.871104 20824 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet 58420a59ee814bf8ad558f693bd460f3 (table rebalancer_dynamic_table [id=430664710eff4ee485a54c3ca3fbfffd]). Expected version 1 got 0
I20240501 20:09:36.873095 24290 tablet.cc:1722] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:36.878273 20824 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 58420a59ee814bf8ad558f693bd460f3 (table rebalancer_dynamic_table [id=430664710eff4ee485a54c3ca3fbfffd]). Expected version 1 got 0
I20240501 20:09:36.885674 24290 tablet.cc:1620] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:36.948128 23691 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:36.949926 21208 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:36.961807 20825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:36.968890 21336 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:09:36.974779 24220 tablet.cc:1722] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:36.975337 24220 tablet.cc:1620] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:36.981711 24287 tablet.cc:1722] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:36.982271 24287 tablet.cc:1620] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:36.984030 24290 tablet.cc:1722] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:36.984520 24290 tablet.cc:1620] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:36.995232 24179 consensus_queue.cc:1035] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.005010 24089 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:37.015430 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=430664710eff4ee485a54c3ca3fbfffd] alter complete (version 2)
I20240501 20:09:37.034261 24179 consensus_queue.cc:1035] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.037451 24224 consensus_queue.cc:1035] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.051803 20945 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:09:37.058526 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:37.084206 23824 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:37.118757 20825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:37.120333 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:09:37.123387 20822 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 20:09:37.123612 20822 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 20:09:37.136065 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 832b77fbbad74123a5c6548732ed43a2
I20240501 20:09:37.136593 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 90c5be8fb35f4298be590564144c5c22
I20240501 20:09:37.137024 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 036f254eaa524900b20c25865f43d401
I20240501 20:09:37.137048 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 832b77fbbad74123a5c6548732ed43a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:37.137928 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 832b77fbbad74123a5c6548732ed43a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:37.138211 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c5be8fb35f4298be590564144c5c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:37.138422 24201 tablet_replica.cc:330] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:37.138645 24201 raft_consensus.cc:2229] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:37.138895 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet 832b77fbbad74123a5c6548732ed43a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:37.138916 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c5be8fb35f4298be590564144c5c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:37.139001 24201 raft_consensus.cc:2258] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.139199 24203 tablet_replica.cc:330] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:37.139405 24203 raft_consensus.cc:2229] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.139566 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c5be8fb35f4298be590564144c5c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:37.139683 24200 tablet_replica.cc:330] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:37.139811 24197 tablet_replica.cc:330] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:37.139811 24201 ts_tablet_manager.cc:1875] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.139909 24200 raft_consensus.cc:2229] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.140033 24197 raft_consensus.cc:2229] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.140129 24200 raft_consensus.cc:2258] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.140249 24197 raft_consensus.cc:2258] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.140823 24200 ts_tablet_manager.cc:1875] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.140873 24197 ts_tablet_manager.cc:1875] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.142840 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 036f254eaa524900b20c25865f43d401 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:37.143100 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 036f254eaa524900b20c25865f43d401 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:37.143617 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 036f254eaa524900b20c25865f43d401 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:37.146066 24203 raft_consensus.cc:2258] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.146800 24203 ts_tablet_manager.cc:1875] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.148334 24201 ts_tablet_manager.cc:1888] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.148537 24201 log.cc:1196] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/832b77fbbad74123a5c6548732ed43a2
I20240501 20:09:37.149055 24201 ts_tablet_manager.cc:1909] T 832b77fbbad74123a5c6548732ed43a2 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:37.152061 24220 tablet.cc:1722] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:37.152726 24197 ts_tablet_manager.cc:1888] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.152861 24197 log.cc:1196] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/90c5be8fb35f4298be590564144c5c22
I20240501 20:09:37.153297 24197 ts_tablet_manager.cc:1909] T 90c5be8fb35f4298be590564144c5c22 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:37.158931 24197 tablet_replica.cc:330] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:37.159049 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 90c5be8fb35f4298be590564144c5c22 (table rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]) successfully deleted
I20240501 20:09:37.159780 24197 raft_consensus.cc:2229] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.160130 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 832b77fbbad74123a5c6548732ed43a2 (table rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]) successfully deleted
I20240501 20:09:37.160239 24201 tablet_replica.cc:330] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:37.160189 24287 tablet.cc:1722] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:37.160472 24201 raft_consensus.cc:2229] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.160823 24201 raft_consensus.cc:2258] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.160620 24197 raft_consensus.cc:2258] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.161543 24201 ts_tablet_manager.cc:1875] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.162238 24197 ts_tablet_manager.cc:1875] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.166213 24201 ts_tablet_manager.cc:1888] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.166338 24201 log.cc:1196] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/90c5be8fb35f4298be590564144c5c22
I20240501 20:09:37.166729 24201 ts_tablet_manager.cc:1909] T 90c5be8fb35f4298be590564144c5c22 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:37.167599 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 90c5be8fb35f4298be590564144c5c22 (table rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]) successfully deleted
I20240501 20:09:37.173377 24290 tablet.cc:1722] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:37.179121 24217 consensus_queue.cc:1035] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.179730 24201 tablet_replica.cc:330] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:37.179955 24201 raft_consensus.cc:2229] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:37.180261 24201 raft_consensus.cc:2258] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.181021 24201 ts_tablet_manager.cc:1875] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.192987 20825 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=430664710eff4ee485a54c3ca3fbfffd] alter complete (version 3)
I20240501 20:09:37.195577 24200 ts_tablet_manager.cc:1888] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.196768 24200 log.cc:1196] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/832b77fbbad74123a5c6548732ed43a2
I20240501 20:09:37.197130 24217 consensus_queue.cc:1035] T e9cc62237cde4a90b386f5c30625445c P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:37.200783 21031 consensus_peers.cc:485] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90c5be8fb35f4298be590564144c5c22. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:37.203653 24224 consensus_queue.cc:1035] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.209348 24201 ts_tablet_manager.cc:1888] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.215958 24201 log.cc:1196] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/036f254eaa524900b20c25865f43d401
I20240501 20:09:37.216637 24201 ts_tablet_manager.cc:1909] T 036f254eaa524900b20c25865f43d401 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:37.216667 24203 ts_tablet_manager.cc:1888] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.218019 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 036f254eaa524900b20c25865f43d401 (table rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]) successfully deleted
I20240501 20:09:37.220758 24197 ts_tablet_manager.cc:1888] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.220883 24197 log.cc:1196] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/036f254eaa524900b20c25865f43d401
I20240501 20:09:37.221340 24197 ts_tablet_manager.cc:1909] T 036f254eaa524900b20c25865f43d401 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:37.222312 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 036f254eaa524900b20c25865f43d401 (table rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]) successfully deleted
I20240501 20:09:37.224849 24203 log.cc:1196] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/832b77fbbad74123a5c6548732ed43a2
I20240501 20:09:37.225800 24203 ts_tablet_manager.cc:1909] T 832b77fbbad74123a5c6548732ed43a2 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:37.226289 24200 ts_tablet_manager.cc:1909] T 832b77fbbad74123a5c6548732ed43a2 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:37.227360 24200 tablet_replica.cc:330] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:37.227394 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 832b77fbbad74123a5c6548732ed43a2 (table rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]) successfully deleted
I20240501 20:09:37.233281 24203 tablet_replica.cc:330] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:37.234405 24200 raft_consensus.cc:2229] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.235055 24200 raft_consensus.cc:2258] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.235671 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 832b77fbbad74123a5c6548732ed43a2 (table rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]) successfully deleted
I20240501 20:09:37.236951 24203 raft_consensus.cc:2229] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:37.237471 24203 raft_consensus.cc:2258] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.239920 24203 ts_tablet_manager.cc:1875] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.240855 24200 ts_tablet_manager.cc:1875] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.270509 24203 ts_tablet_manager.cc:1888] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.270826 24203 log.cc:1196] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/90c5be8fb35f4298be590564144c5c22
I20240501 20:09:37.271274 24203 ts_tablet_manager.cc:1909] T 90c5be8fb35f4298be590564144c5c22 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:37.273197 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 90c5be8fb35f4298be590564144c5c22 (table rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]) successfully deleted
I20240501 20:09:37.275382 24200 ts_tablet_manager.cc:1888] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.275506 24200 log.cc:1196] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/036f254eaa524900b20c25865f43d401
I20240501 20:09:37.276018 24200 ts_tablet_manager.cc:1909] T 036f254eaa524900b20c25865f43d401 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:37.277057 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 036f254eaa524900b20c25865f43d401 (table rebalance_test_table_4 [id=2a32411dcbfb4c1484dd2295ef5909d2]) successfully deleted
I20240501 20:09:37.296658 20825 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:37.305508 20825 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:37.305995 20825 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:37.306345 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 2ded048b4bb94eb79f95554db7e15b7b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.306810 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ded048b4bb94eb79f95554db7e15b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.308800 20825 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 58420a59ee814bf8ad558f693bd460f3
I20240501 20:09:37.309525 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 58420a59ee814bf8ad558f693bd460f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:37.309758 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 58420a59ee814bf8ad558f693bd460f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:37.309826 24197 tablet_replica.cc:330] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:37.310070 24197 raft_consensus.cc:2229] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.310253 24197 raft_consensus.cc:2258] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.311498 24197 ts_tablet_manager.cc:1875] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.312331 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 58420a59ee814bf8ad558f693bd460f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:37.312446 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 2ded048b4bb94eb79f95554db7e15b7b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.312616 24201 tablet_replica.cc:330] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:37.312810 24201 raft_consensus.cc:2229] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.312839 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ded048b4bb94eb79f95554db7e15b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.313055 24201 raft_consensus.cc:2258] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.316365 24201 ts_tablet_manager.cc:1875] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.316430 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 2ded048b4bb94eb79f95554db7e15b7b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.316843 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ded048b4bb94eb79f95554db7e15b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.317449 24197 ts_tablet_manager.cc:1888] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:37.317572 24197 log.cc:1196] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/58420a59ee814bf8ad558f693bd460f3
I20240501 20:09:37.317970 24197 ts_tablet_manager.cc:1909] T 58420a59ee814bf8ad558f693bd460f3 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:37.319417 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 58420a59ee814bf8ad558f693bd460f3 (table rebalancer_dynamic_table_new_name [id=430664710eff4ee485a54c3ca3fbfffd]) successfully deleted
I20240501 20:09:37.321974 23826 tablet_service.cc:1467] Processing CreateTablet for tablet cdb92027f1be4807a83c907ee32bfab9 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.322376 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb92027f1be4807a83c907ee32bfab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.324471 24201 ts_tablet_manager.cc:1888] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:37.324620 24201 log.cc:1196] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/58420a59ee814bf8ad558f693bd460f3
I20240501 20:09:37.325135 24201 ts_tablet_manager.cc:1909] T 58420a59ee814bf8ad558f693bd460f3 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:37.326309 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 58420a59ee814bf8ad558f693bd460f3 (table rebalancer_dynamic_table_new_name [id=430664710eff4ee485a54c3ca3fbfffd]) successfully deleted
I20240501 20:09:37.326938 23956 tablet_service.cc:1467] Processing CreateTablet for tablet cdb92027f1be4807a83c907ee32bfab9 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.327409 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb92027f1be4807a83c907ee32bfab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.328945 24322 tablet_replica.cc:330] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:37.329123 24322 raft_consensus.cc:2229] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:37.329595 24322 raft_consensus.cc:2258] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.330369 24322 ts_tablet_manager.cc:1875] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.334139 21076 tablet_service.cc:1467] Processing CreateTablet for tablet cdb92027f1be4807a83c907ee32bfab9 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.334611 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb92027f1be4807a83c907ee32bfab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.335170 24322 ts_tablet_manager.cc:1888] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:37.335253 24322 log.cc:1196] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/58420a59ee814bf8ad558f693bd460f3
I20240501 20:09:37.335577 24322 ts_tablet_manager.cc:1909] T 58420a59ee814bf8ad558f693bd460f3 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:37.336548 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 58420a59ee814bf8ad558f693bd460f3 (table rebalancer_dynamic_table_new_name [id=430664710eff4ee485a54c3ca3fbfffd]) successfully deleted
I20240501 20:09:37.341902 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 236ba60733e1416395346179be3b3888 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.342383 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236ba60733e1416395346179be3b3888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.342245 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 236ba60733e1416395346179be3b3888 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.342820 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236ba60733e1416395346179be3b3888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.343601 24323 tablet_bootstrap.cc:492] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:37.345599 24323 tablet_bootstrap.cc:654] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.356026 24323 tablet_bootstrap.cc:492] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:37.356535 24323 ts_tablet_manager.cc:1367] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.005s
I20240501 20:09:37.356608 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 236ba60733e1416395346179be3b3888 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.357164 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236ba60733e1416395346179be3b3888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.358101 24323 raft_consensus.cc:348] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:37.359102 24324 tablet_bootstrap.cc:492] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:37.359500 24323 raft_consensus.cc:374] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.361606 24324 tablet_bootstrap.cc:654] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.361799 24323 raft_consensus.cc:729] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.362248 24323 consensus_queue.cc:260] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:37.363994 24323 ts_tablet_manager.cc:1398] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:09:37.364331 24323 tablet_bootstrap.cc:492] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:37.367905 24328 tablet_bootstrap.cc:492] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:37.369824 24323 tablet_bootstrap.cc:654] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.369941 24329 tablet_bootstrap.cc:492] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:37.370023 24328 tablet_bootstrap.cc:654] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.370919 24326 tablet_bootstrap.cc:492] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:37.371335 24327 tablet_bootstrap.cc:492] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:37.372901 24329 tablet_bootstrap.cc:654] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.373185 24327 tablet_bootstrap.cc:654] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.374939 24326 tablet_bootstrap.cc:654] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.375792 24323 tablet_bootstrap.cc:492] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:37.376013 24323 ts_tablet_manager.cc:1367] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20240501 20:09:37.376909 24323 raft_consensus.cc:348] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.377207 24323 raft_consensus.cc:374] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.377344 24323 raft_consensus.cc:729] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.377697 24323 consensus_queue.cc:260] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.378989 24323 ts_tablet_manager.cc:1398] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:37.380093 24328 tablet_bootstrap.cc:492] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:37.380234 24328 ts_tablet_manager.cc:1367] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 20:09:37.381002 24328 raft_consensus.cc:348] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:37.381230 24328 raft_consensus.cc:374] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.381332 24328 raft_consensus.cc:729] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.381552 24328 consensus_queue.cc:260] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:37.385577 24328 ts_tablet_manager.cc:1398] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:37.391045 24329 tablet_bootstrap.cc:492] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:37.391116 24324 tablet_bootstrap.cc:492] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:37.391234 24329 ts_tablet_manager.cc:1367] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20240501 20:09:37.391322 24324 ts_tablet_manager.cc:1367] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240501 20:09:37.392304 24329 raft_consensus.cc:348] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:37.392597 24329 raft_consensus.cc:374] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.392705 24329 raft_consensus.cc:729] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.392489 24324 raft_consensus.cc:348] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:37.393062 24324 raft_consensus.cc:374] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.393023 24329 consensus_queue.cc:260] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:37.393230 24324 raft_consensus.cc:729] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.394042 24324 consensus_queue.cc:260] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:37.395025 24326 tablet_bootstrap.cc:492] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:37.395897 24324 ts_tablet_manager.cc:1398] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:37.396850 24326 ts_tablet_manager.cc:1367] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240501 20:09:37.397702 24326 raft_consensus.cc:348] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:37.398497 24326 raft_consensus.cc:374] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.398653 24326 raft_consensus.cc:729] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.399317 24327 tablet_bootstrap.cc:492] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:37.399936 24327 ts_tablet_manager.cc:1367] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.014s
I20240501 20:09:37.400789 24327 raft_consensus.cc:348] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.401206 24327 raft_consensus.cc:374] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.401336 24327 raft_consensus.cc:729] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.401650 24327 consensus_queue.cc:260] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.402393 24327 ts_tablet_manager.cc:1398] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:37.404821 24329 ts_tablet_manager.cc:1398] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:09:37.406730 24326 consensus_queue.cc:260] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:37.408574 24326 ts_tablet_manager.cc:1398] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 20:09:37.410470 24311 raft_consensus.cc:482] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.410636 24311 raft_consensus.cc:504] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:37.411708 24311 leader_election.cc:288] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:37.414958 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb92027f1be4807a83c907ee32bfab9" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:09:37.415448 23912 leader_election.cc:341] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:37.418706 24326 tablet_bootstrap.cc:492] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:37.420133 24326 tablet_bootstrap.cc:654] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.420970 20822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:37.423169 24326 tablet_bootstrap.cc:492] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:37.423382 24326 ts_tablet_manager.cc:1367] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240501 20:09:37.424155 24326 raft_consensus.cc:348] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:37.424417 24326 raft_consensus.cc:374] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.424530 24326 raft_consensus.cc:729] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.424836 24326 consensus_queue.cc:260] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:37.426026 24326 ts_tablet_manager.cc:1398] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 20:09:37.426332 24326 tablet_bootstrap.cc:492] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:37.427944 24326 tablet_bootstrap.cc:654] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.430322 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb92027f1be4807a83c907ee32bfab9" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:37.430562 21097 raft_consensus.cc:2454] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:37.431198 23909 leader_election.cc:302] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, feac73db0a3b4aafa374ca5dfd322b08; no voters: c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:37.431645 24311 raft_consensus.cc:2781] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.431778 24311 raft_consensus.cc:482] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.431869 24311 raft_consensus.cc:3037] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.434959 24311 raft_consensus.cc:504] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:37.436846 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb92027f1be4807a83c907ee32bfab9" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:37.437132 21097 raft_consensus.cc:3037] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.438779 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 694fe13e19d54d73804f696f1fd394bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8e08fc46ecf84e308d499d27f9b299e8]), partition=
I20240501 20:09:37.439184 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694fe13e19d54d73804f696f1fd394bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.441824 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 694fe13e19d54d73804f696f1fd394bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8e08fc46ecf84e308d499d27f9b299e8]), partition=
I20240501 20:09:37.442322 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694fe13e19d54d73804f696f1fd394bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.443142 21097 raft_consensus.cc:2454] T cdb92027f1be4807a83c907ee32bfab9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:37.443588 23909 leader_election.cc:302] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:37.445144 24328 tablet_bootstrap.cc:492] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:37.447190 24328 tablet_bootstrap.cc:654] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.452679 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 694fe13e19d54d73804f696f1fd394bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8e08fc46ecf84e308d499d27f9b299e8]), partition=
I20240501 20:09:37.453086 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694fe13e19d54d73804f696f1fd394bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.457348 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb92027f1be4807a83c907ee32bfab9" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:37.457585 23846 raft_consensus.cc:3037] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.460624 23846 raft_consensus.cc:2454] T cdb92027f1be4807a83c907ee32bfab9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:37.462744 24346 tablet_bootstrap.cc:492] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:37.465138 24346 tablet_bootstrap.cc:654] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.465945 24311 leader_election.cc:288] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:37.466806 24311 raft_consensus.cc:2781] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.467773 24311 raft_consensus.cc:686] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:37.470054 24171 raft_consensus.cc:482] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.470242 24171 raft_consensus.cc:504] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:37.471418 24171 leader_election.cc:288] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:37.477012 24327 tablet_bootstrap.cc:492] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:37.478559 24327 tablet_bootstrap.cc:654] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.487843 24327 tablet_bootstrap.cc:492] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:37.488036 24327 ts_tablet_manager.cc:1367] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20240501 20:09:37.488883 24327 raft_consensus.cc:348] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.489121 24327 raft_consensus.cc:374] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.489213 24327 raft_consensus.cc:729] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.489485 24327 consensus_queue.cc:260] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.490015 24311 consensus_queue.cc:237] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:37.490775 24327 ts_tablet_manager.cc:1398] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240501 20:09:37.488390 23909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 44801.8us
I20240501 20:09:37.490406 24328 tablet_bootstrap.cc:492] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:37.492305 24328 ts_tablet_manager.cc:1367] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.003s
I20240501 20:09:37.494514 24328 raft_consensus.cc:348] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.496749 24328 raft_consensus.cc:374] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.496907 24328 raft_consensus.cc:729] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.497427 24328 consensus_queue.cc:260] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.497970 24328 ts_tablet_manager.cc:1398] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 20:09:37.507061 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236ba60733e1416395346179be3b3888" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:37.507386 23711 raft_consensus.cc:2454] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:37.512720 21293 leader_election.cc:302] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:37.513265 23629 raft_consensus.cc:2781] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.513377 23629 raft_consensus.cc:482] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.513494 23629 raft_consensus.cc:3037] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.527580 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236ba60733e1416395346179be3b3888" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:37.528028 23846 raft_consensus.cc:2454] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:37.517730 23629 raft_consensus.cc:504] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:37.539903 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236ba60733e1416395346179be3b3888" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:37.540223 23711 raft_consensus.cc:3037] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.543367 24346 tablet_bootstrap.cc:492] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:37.543608 24346 ts_tablet_manager.cc:1367] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.006s
I20240501 20:09:37.544407 24346 raft_consensus.cc:348] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.544636 24346 raft_consensus.cc:374] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.544724 24346 raft_consensus.cc:729] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.544955 24346 consensus_queue.cc:260] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.545486 24346 ts_tablet_manager.cc:1398] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:37.547806 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236ba60733e1416395346179be3b3888" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:37.548231 23846 raft_consensus.cc:3037] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.552850 23711 raft_consensus.cc:2454] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:37.556239 21293 leader_election.cc:302] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:37.559671 24171 raft_consensus.cc:2781] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.575826 23629 leader_election.cc:288] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:37.589885 24356 raft_consensus.cc:482] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.590158 24356 raft_consensus.cc:504] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.591676 24356 leader_election.cc:288] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:37.594064 24217 raft_consensus.cc:482] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.594242 24217 raft_consensus.cc:504] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.594940 24217 leader_election.cc:288] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:37.595068 23846 raft_consensus.cc:2454] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
W20240501 20:09:37.606182 21404 tablet.cc:2369] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:37.622352 24171 raft_consensus.cc:686] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:37.622408 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fe13e19d54d73804f696f1fd394bc" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:37.622642 21225 raft_consensus.cc:2454] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:37.622730 24171 consensus_queue.cc:237] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:37.623062 21294 leader_election.cc:302] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:37.623404 24355 raft_consensus.cc:2781] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.623492 24355 raft_consensus.cc:482] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.623646 24355 raft_consensus.cc:3037] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.627782 24355 raft_consensus.cc:504] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.628882 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fe13e19d54d73804f696f1fd394bc" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:37.629097 21225 raft_consensus.cc:3037] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.631747 24326 tablet_bootstrap.cc:492] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:37.632133 24326 ts_tablet_manager.cc:1367] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.206s user 0.004s sys 0.003s
I20240501 20:09:37.632373 21225 raft_consensus.cc:2454] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:37.632722 21294 leader_election.cc:302] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:37.632972 24356 raft_consensus.cc:2781] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.633176 24356 raft_consensus.cc:686] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:37.633270 24355 leader_election.cc:288] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:37.633352 24326 raft_consensus.cc:348] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.633574 24326 raft_consensus.cc:374] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.633471 24356 consensus_queue.cc:237] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.633656 24326 raft_consensus.cc:729] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.634096 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fe13e19d54d73804f696f1fd394bc" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:37.633908 24326 consensus_queue.cc:260] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.634326 24110 raft_consensus.cc:2454] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:37.634868 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fe13e19d54d73804f696f1fd394bc" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:37.635040 24110 raft_consensus.cc:3037] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.637732 24110 raft_consensus.cc:2454] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:37.641058 20825 catalog_manager.cc:5484] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:37.642715 24326 ts_tablet_manager.cc:1398] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:09:37.642982 20826 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 20:09:37.643230 20826 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 20:09:37.646844 20822 catalog_manager.cc:5484] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:37.653951 20826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet be7bba5002cc4c72b46884ef394a3d15
I20240501 20:09:37.654868 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet be7bba5002cc4c72b46884ef394a3d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:37.655259 24203 tablet_replica.cc:330] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:37.655583 24203 raft_consensus.cc:2229] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:37.655939 20826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 9845d5032786442e8da848ebf3b111dd
I20240501 20:09:37.656019 24203 raft_consensus.cc:2258] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.656124 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet be7bba5002cc4c72b46884ef394a3d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:37.656392 24201 tablet_replica.cc:330] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:37.656620 24201 raft_consensus.cc:2229] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.656791 24203 ts_tablet_manager.cc:1875] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.656850 24201 raft_consensus.cc:2258] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.657083 20826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 493824843c024d50b1a4698ca13d2089
I20240501 20:09:37.657528 24201 ts_tablet_manager.cc:1875] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.658610 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 493824843c024d50b1a4698ca13d2089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:37.659680 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet 493824843c024d50b1a4698ca13d2089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:37.659953 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 9845d5032786442e8da848ebf3b111dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:37.660122 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 9845d5032786442e8da848ebf3b111dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:37.660324 24197 tablet_replica.cc:330] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:37.660382 24200 tablet_replica.cc:330] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:37.660413 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 9845d5032786442e8da848ebf3b111dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:37.660584 24197 raft_consensus.cc:2229] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.660614 24200 raft_consensus.cc:2229] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.660831 24200 raft_consensus.cc:2258] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.660864 24197 raft_consensus.cc:2258] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.661546 24200 ts_tablet_manager.cc:1875] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.661782 24197 ts_tablet_manager.cc:1875] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.659969 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet be7bba5002cc4c72b46884ef394a3d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:37.664728 24201 ts_tablet_manager.cc:1888] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.664891 24201 log.cc:1196] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/be7bba5002cc4c72b46884ef394a3d15
I20240501 20:09:37.665313 24201 ts_tablet_manager.cc:1909] T be7bba5002cc4c72b46884ef394a3d15 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:37.666244 24201 tablet_replica.cc:330] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:37.666378 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet be7bba5002cc4c72b46884ef394a3d15 (table rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]) successfully deleted
I20240501 20:09:37.666438 24201 raft_consensus.cc:2229] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.666654 24201 raft_consensus.cc:2258] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.667297 24201 ts_tablet_manager.cc:1875] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.670799 24200 ts_tablet_manager.cc:1888] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.670972 24200 log.cc:1196] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/9845d5032786442e8da848ebf3b111dd
I20240501 20:09:37.671422 24200 ts_tablet_manager.cc:1909] T 9845d5032786442e8da848ebf3b111dd P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:37.671788 24203 ts_tablet_manager.cc:1888] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.671898 24203 log.cc:1196] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/be7bba5002cc4c72b46884ef394a3d15
I20240501 20:09:37.672299 24203 ts_tablet_manager.cc:1909] T be7bba5002cc4c72b46884ef394a3d15 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:37.673827 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 9845d5032786442e8da848ebf3b111dd (table rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]) successfully deleted
I20240501 20:09:37.676410 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 493824843c024d50b1a4698ca13d2089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:37 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:37.677381 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet be7bba5002cc4c72b46884ef394a3d15 (table rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]) successfully deleted
I20240501 20:09:37.677640 24203 tablet_replica.cc:330] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:37.677889 24203 raft_consensus.cc:2229] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:37.678239 24203 raft_consensus.cc:2258] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.680997 24201 ts_tablet_manager.cc:1888] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.681962 24201 log.cc:1196] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/493824843c024d50b1a4698ca13d2089
I20240501 20:09:37.682632 24201 ts_tablet_manager.cc:1909] T 493824843c024d50b1a4698ca13d2089 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:37.682838 24197 ts_tablet_manager.cc:1888] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.683039 24197 log.cc:1196] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/9845d5032786442e8da848ebf3b111dd
I20240501 20:09:37.683507 24197 ts_tablet_manager.cc:1909] T 9845d5032786442e8da848ebf3b111dd P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:37.684273 24203 ts_tablet_manager.cc:1875] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.684249 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 493824843c024d50b1a4698ca13d2089 (table rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]) successfully deleted
I20240501 20:09:37.684991 24201 tablet_replica.cc:330] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:37.685292 24201 raft_consensus.cc:2229] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:37.684007 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fe13e19d54d73804f696f1fd394bc" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:37.685611 21225 raft_consensus.cc:2379] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8559229e044b4e1ea661b881b3f9af50 in current term 1: Already voted for candidate 61380023001a4ee9ada9ec2c6c1b7159 in this term.
I20240501 20:09:37.685690 24201 raft_consensus.cc:2258] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.688258 24201 ts_tablet_manager.cc:1875] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.689260 20826 catalog_manager.cc:5484] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:37.694357 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 9845d5032786442e8da848ebf3b111dd (table rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]) successfully deleted
I20240501 20:09:37.698546 24201 ts_tablet_manager.cc:1888] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.698745 24201 log.cc:1196] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/9845d5032786442e8da848ebf3b111dd
I20240501 20:09:37.699302 24201 ts_tablet_manager.cc:1909] T 9845d5032786442e8da848ebf3b111dd P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:37.699534 24197 tablet_replica.cc:330] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:37.699784 24197 raft_consensus.cc:2229] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.702754 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 9845d5032786442e8da848ebf3b111dd (table rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]) successfully deleted
I20240501 20:09:37.702967 24203 ts_tablet_manager.cc:1888] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.703215 24197 raft_consensus.cc:2258] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.703657 24203 log.cc:1196] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/493824843c024d50b1a4698ca13d2089
I20240501 20:09:37.704108 24203 ts_tablet_manager.cc:1909] T 493824843c024d50b1a4698ca13d2089 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:37.705003 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 493824843c024d50b1a4698ca13d2089 (table rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]) successfully deleted
I20240501 20:09:37.704121 24197 ts_tablet_manager.cc:1875] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.710248 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694fe13e19d54d73804f696f1fd394bc" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:37.710888 24044 leader_election.cc:302] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:09:37.711359 24217 raft_consensus.cc:2726] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:37.728322 24197 ts_tablet_manager.cc:1888] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.728466 24197 log.cc:1196] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/be7bba5002cc4c72b46884ef394a3d15
I20240501 20:09:37.728881 24197 ts_tablet_manager.cc:1909] T be7bba5002cc4c72b46884ef394a3d15 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:37.734200 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet be7bba5002cc4c72b46884ef394a3d15 (table rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]) successfully deleted
I20240501 20:09:37.734692 24197 tablet_replica.cc:330] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:37.734894 24197 raft_consensus.cc:2229] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:37.735765 24197 raft_consensus.cc:2258] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:37.736379 24197 ts_tablet_manager.cc:1875] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:37.742426 24197 ts_tablet_manager.cc:1888] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:37.742543 24197 log.cc:1196] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/493824843c024d50b1a4698ca13d2089
I20240501 20:09:37.742861 24197 ts_tablet_manager.cc:1909] T 493824843c024d50b1a4698ca13d2089 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:37.743649 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 493824843c024d50b1a4698ca13d2089 (table rebalance_test_table_6 [id=a77b1887d5e5490398d2f1073c62a2a8]) successfully deleted
I20240501 20:09:37.775346 24217 raft_consensus.cc:482] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.775575 24217 raft_consensus.cc:504] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.776024 24311 raft_consensus.cc:482] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.776300 24217 leader_election.cc:288] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:37.776273 24311 raft_consensus.cc:504] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.776748 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded048b4bb94eb79f95554db7e15b7b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:37.776921 24311 leader_election.cc:288] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:37.776991 23977 raft_consensus.cc:2454] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:37.776962 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded048b4bb94eb79f95554db7e15b7b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:37.777208 23846 raft_consensus.cc:2454] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:37.777571 24043 leader_election.cc:302] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:37.777769 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded048b4bb94eb79f95554db7e15b7b" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:37.777812 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded048b4bb94eb79f95554db7e15b7b" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:37.777968 23846 raft_consensus.cc:2454] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:37.778023 24110 raft_consensus.cc:2454] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:37.778338 24217 raft_consensus.cc:2781] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.778323 23912 leader_election.cc:302] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:37.778424 24217 raft_consensus.cc:482] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.778517 24217 raft_consensus.cc:3037] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.778813 24311 raft_consensus.cc:2781] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.778914 24311 raft_consensus.cc:482] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.779014 24311 raft_consensus.cc:3037] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.781664 24311 raft_consensus.cc:504] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.781373 24217 raft_consensus.cc:504] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.782680 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded048b4bb94eb79f95554db7e15b7b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:37.782781 24311 leader_election.cc:288] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:37.782845 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded048b4bb94eb79f95554db7e15b7b" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:37.782946 23977 raft_consensus.cc:2379] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8559229e044b4e1ea661b881b3f9af50 in current term 1: Already voted for candidate 744c5a532e724391830c0785d32eb2fb in this term.
I20240501 20:09:37.783088 24110 raft_consensus.cc:2379] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 744c5a532e724391830c0785d32eb2fb in current term 1: Already voted for candidate 8559229e044b4e1ea661b881b3f9af50 in this term.
I20240501 20:09:37.783615 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded048b4bb94eb79f95554db7e15b7b" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:37.783802 23846 raft_consensus.cc:3037] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.783919 24217 leader_election.cc:288] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:37.784193 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ded048b4bb94eb79f95554db7e15b7b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:37.784737 24046 leader_election.cc:302] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50; no voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:37.785135 24217 raft_consensus.cc:2726] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:09:37.786224 23846 raft_consensus.cc:2454] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:37.786646 23912 leader_election.cc:302] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:37.786919 24311 raft_consensus.cc:2781] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.787089 24311 raft_consensus.cc:686] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:37.787346 24311 consensus_queue.cc:237] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:37.790501 20826 catalog_manager.cc:5484] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:37.823606 20826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:37.834645 21225 raft_consensus.cc:1261] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:37.836807 24110 raft_consensus.cc:1261] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:37.837996 24356 consensus_queue.cc:1035] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.841517 24171 consensus_queue.cc:1035] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.874920 20826 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 694fe13e19d54d73804f696f1fd394bc (table rebalancer_dynamic_table [id=8e08fc46ecf84e308d499d27f9b299e8]). Expected version 1 got 0
I20240501 20:09:37.876787 24411 tablet.cc:1722] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:37.877915 24411 tablet.cc:1620] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:37.878926 20818 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet 694fe13e19d54d73804f696f1fd394bc (table rebalancer_dynamic_table [id=8e08fc46ecf84e308d499d27f9b299e8]). Expected version 1 got 0
I20240501 20:09:37.885388 24410 tablet.cc:1722] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:37.886652 24410 tablet.cc:1620] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: MemRowSet was empty: no flush needed.
I20240501 20:09:37.887048 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=8e08fc46ecf84e308d499d27f9b299e8] alter complete (version 1)
I20240501 20:09:37.887718 24412 tablet.cc:1722] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:37.889358 24412 tablet.cc:1620] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:37.897640 20822 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 694fe13e19d54d73804f696f1fd394bc (table rebalancer_dynamic_table [id=8e08fc46ecf84e308d499d27f9b299e8]). Expected version 1 got 0
I20240501 20:09:37.939250 23691 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:37.946189 21206 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:37.949012 21336 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:09:37.989943 23956 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:37.991854 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:38.006384 24089 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:38.009853 24410 tablet.cc:1722] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:38.010356 24410 tablet.cc:1620] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: MemRowSet was empty: no flush needed.
I20240501 20:09:38.016782 24411 tablet.cc:1722] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:38.017284 24411 tablet.cc:1620] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:38.020568 24412 tablet.cc:1722] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:38.021026 24412 tablet.cc:1620] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:38.026947 24311 consensus_queue.cc:1035] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.029206 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=8e08fc46ecf84e308d499d27f9b299e8] alter complete (version 2)
I20240501 20:09:38.031505 23825 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:38.065796 24344 consensus_queue.cc:1035] T cdb92027f1be4807a83c907ee32bfab9 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.078188 24171 consensus_queue.cc:1035] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.094959 20945 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:09:38.094938 24171 consensus_queue.cc:1035] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.137655 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:38.147786 24410 tablet.cc:1722] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:38.149044 24411 tablet.cc:1722] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:38.152379 24412 tablet.cc:1722] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:38.162708 20821 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 20:09:38.163023 20821 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 20:09:38.163933 20826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=8e08fc46ecf84e308d499d27f9b299e8] alter complete (version 3)
I20240501 20:09:38.169119 20821 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet f552dfb425734b6f943914725274d1e4
I20240501 20:09:38.170167 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet f552dfb425734b6f943914725274d1e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:38.170192 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet f552dfb425734b6f943914725274d1e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:38.170452 24201 tablet_replica.cc:330] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:38.170557 24203 tablet_replica.cc:330] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:38.170631 24201 raft_consensus.cc:2229] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:38.170773 24203 raft_consensus.cc:2229] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.170981 24201 raft_consensus.cc:2258] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.171023 24203 raft_consensus.cc:2258] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.171728 24201 ts_tablet_manager.cc:1875] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.172660 24203 ts_tablet_manager.cc:1875] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.180222 20821 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 85be3407c8724e7d89e1f4df16808e9a
I20240501 20:09:38.180287 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet f552dfb425734b6f943914725274d1e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:38.180966 20821 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet f999e1ac85544f228dad6e422352d759
I20240501 20:09:38.181521 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 85be3407c8724e7d89e1f4df16808e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:38.181609 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 85be3407c8724e7d89e1f4df16808e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:38.181929 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 85be3407c8724e7d89e1f4df16808e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:38.182101 21337 tablet_service.cc:1514] Processing DeleteTablet for tablet f999e1ac85544f228dad6e422352d759 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:38.182202 24197 tablet_replica.cc:330] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:38.182423 24197 raft_consensus.cc:2229] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.182667 24197 raft_consensus.cc:2258] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.182802 24201 ts_tablet_manager.cc:1888] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.182955 24201 log.cc:1196] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/f552dfb425734b6f943914725274d1e4
I20240501 20:09:38.183503 24201 ts_tablet_manager.cc:1909] T f552dfb425734b6f943914725274d1e4 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:38.183503 24197 ts_tablet_manager.cc:1875] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.183606 24203 ts_tablet_manager.cc:1888] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.183724 24203 log.cc:1196] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/f552dfb425734b6f943914725274d1e4
I20240501 20:09:38.184191 24203 ts_tablet_manager.cc:1909] T f552dfb425734b6f943914725274d1e4 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:38.184653 24201 tablet_replica.cc:330] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:38.184937 24201 raft_consensus.cc:2229] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.185217 24201 raft_consensus.cc:2258] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.185391 24203 tablet_replica.cc:330] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:38.185601 24203 raft_consensus.cc:2229] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:38.185922 24203 raft_consensus.cc:2258] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.186110 24201 ts_tablet_manager.cc:1875] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.188881 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet f552dfb425734b6f943914725274d1e4 (table rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]) successfully deleted
I20240501 20:09:38.189111 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet f552dfb425734b6f943914725274d1e4 (table rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]) successfully deleted
I20240501 20:09:38.189208 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet f999e1ac85544f228dad6e422352d759 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:38.190088 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet f999e1ac85544f228dad6e422352d759 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:38.192117 24203 ts_tablet_manager.cc:1875] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.194031 24197 ts_tablet_manager.cc:1888] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.194175 24197 log.cc:1196] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/85be3407c8724e7d89e1f4df16808e9a
I20240501 20:09:38.194180 24433 tablet_replica.cc:330] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:38.194402 24433 raft_consensus.cc:2229] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.194693 24433 raft_consensus.cc:2258] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.195487 24433 ts_tablet_manager.cc:1875] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.196053 24197 ts_tablet_manager.cc:1909] T 85be3407c8724e7d89e1f4df16808e9a P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:38.196959 24197 tablet_replica.cc:330] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:38.197037 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 85be3407c8724e7d89e1f4df16808e9a (table rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]) successfully deleted
I20240501 20:09:38.197160 24197 raft_consensus.cc:2229] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.197365 24197 raft_consensus.cc:2258] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.198006 24197 ts_tablet_manager.cc:1875] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.200618 21076 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:38.202193 24201 ts_tablet_manager.cc:1888] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.202416 24201 log.cc:1196] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/85be3407c8724e7d89e1f4df16808e9a
I20240501 20:09:38.202957 24201 ts_tablet_manager.cc:1909] T 85be3407c8724e7d89e1f4df16808e9a P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:38.204610 24201 tablet_replica.cc:330] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:38.204643 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 85be3407c8724e7d89e1f4df16808e9a (table rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]) successfully deleted
I20240501 20:09:38.206482 24433 ts_tablet_manager.cc:1888] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.206749 24201 raft_consensus.cc:2229] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:38.207149 24201 raft_consensus.cc:2258] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.207935 24201 ts_tablet_manager.cc:1875] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.210582 24433 log.cc:1196] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/f552dfb425734b6f943914725274d1e4
I20240501 20:09:38.211201 24433 ts_tablet_manager.cc:1909] T f552dfb425734b6f943914725274d1e4 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:38.215448 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet f552dfb425734b6f943914725274d1e4 (table rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]) successfully deleted
I20240501 20:09:38.215574 24433 tablet_replica.cc:330] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:38.215795 24433 raft_consensus.cc:2229] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.216037 24433 raft_consensus.cc:2258] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.216780 24433 ts_tablet_manager.cc:1875] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.218295 24201 ts_tablet_manager.cc:1888] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.218497 24201 log.cc:1196] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/f999e1ac85544f228dad6e422352d759
I20240501 20:09:38.219017 24201 ts_tablet_manager.cc:1909] T f999e1ac85544f228dad6e422352d759 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:38.220031 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet f999e1ac85544f228dad6e422352d759 (table rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]) successfully deleted
I20240501 20:09:38.222581 24433 ts_tablet_manager.cc:1888] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.222759 24433 log.cc:1196] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/f999e1ac85544f228dad6e422352d759
I20240501 20:09:38.223284 24433 ts_tablet_manager.cc:1909] T f999e1ac85544f228dad6e422352d759 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:38.224481 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet f999e1ac85544f228dad6e422352d759 (table rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]) successfully deleted
I20240501 20:09:38.226621 24203 ts_tablet_manager.cc:1888] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.226817 24203 log.cc:1196] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/85be3407c8724e7d89e1f4df16808e9a
I20240501 20:09:38.227427 24203 ts_tablet_manager.cc:1909] T 85be3407c8724e7d89e1f4df16808e9a P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:38.228453 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 85be3407c8724e7d89e1f4df16808e9a (table rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]) successfully deleted
I20240501 20:09:38.231170 24197 ts_tablet_manager.cc:1888] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.231304 24197 log.cc:1196] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/f999e1ac85544f228dad6e422352d759
I20240501 20:09:38.231753 24197 ts_tablet_manager.cc:1909] T f999e1ac85544f228dad6e422352d759 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:38.232769 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet f999e1ac85544f228dad6e422352d759 (table rebalance_test_table_8 [id=3ad9566df81045c79109411d5d5bf7f0]) successfully deleted
I20240501 20:09:38.269029 20823 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:38.269321 20823 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:38.273432 20823 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 694fe13e19d54d73804f696f1fd394bc
I20240501 20:09:38.274734 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 694fe13e19d54d73804f696f1fd394bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:38.274857 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 694fe13e19d54d73804f696f1fd394bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:38.275172 24197 tablet_replica.cc:330] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:38.275305 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 694fe13e19d54d73804f696f1fd394bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:38.275404 24197 raft_consensus.cc:2229] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.275599 24433 tablet_replica.cc:330] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:38.275614 24439 tablet_replica.cc:330] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:38.275652 24197 raft_consensus.cc:2258] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.275801 24433 raft_consensus.cc:2229] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:38.276150 24433 raft_consensus.cc:2258] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.276167 24439 raft_consensus.cc:2229] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.276458 24439 raft_consensus.cc:2258] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.277069 24197 ts_tablet_manager.cc:1875] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.278020 24439 ts_tablet_manager.cc:1875] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.283532 24197 ts_tablet_manager.cc:1888] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:38.283887 24197 log.cc:1196] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/694fe13e19d54d73804f696f1fd394bc
I20240501 20:09:38.284339 24433 ts_tablet_manager.cc:1875] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.284457 24197 ts_tablet_manager.cc:1909] T 694fe13e19d54d73804f696f1fd394bc P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:38.284478 24439 ts_tablet_manager.cc:1888] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:38.284627 24439 log.cc:1196] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/694fe13e19d54d73804f696f1fd394bc
I20240501 20:09:38.285163 24439 ts_tablet_manager.cc:1909] T 694fe13e19d54d73804f696f1fd394bc P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:38.285835 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 694fe13e19d54d73804f696f1fd394bc (table rebalancer_dynamic_table_new_name [id=8e08fc46ecf84e308d499d27f9b299e8]) successfully deleted
I20240501 20:09:38.287461 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 694fe13e19d54d73804f696f1fd394bc (table rebalancer_dynamic_table_new_name [id=8e08fc46ecf84e308d499d27f9b299e8]) successfully deleted
I20240501 20:09:38.289948 24433 ts_tablet_manager.cc:1888] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:38.290067 24433 log.cc:1196] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/694fe13e19d54d73804f696f1fd394bc
I20240501 20:09:38.290539 24433 ts_tablet_manager.cc:1909] T 694fe13e19d54d73804f696f1fd394bc P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:38.293020 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 694fe13e19d54d73804f696f1fd394bc (table rebalancer_dynamic_table_new_name [id=8e08fc46ecf84e308d499d27f9b299e8]) successfully deleted
I20240501 20:09:38.334545 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:38.343423 23956 tablet_service.cc:1467] Processing CreateTablet for tablet a9fd7894fc5745ae9588437cd5bd0265 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:38.343935 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fd7894fc5745ae9588437cd5bd0265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.343853 23691 tablet_service.cc:1467] Processing CreateTablet for tablet a9fd7894fc5745ae9588437cd5bd0265 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:38.344272 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fd7894fc5745ae9588437cd5bd0265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.347867 23690 tablet_service.cc:1467] Processing CreateTablet for tablet 30eb2ed6c8b64204b4c3a11b41e53ec8 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:38.348317 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30eb2ed6c8b64204b4c3a11b41e53ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.348790 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 30eb2ed6c8b64204b4c3a11b41e53ec8 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:38.349241 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30eb2ed6c8b64204b4c3a11b41e53ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.349403 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 30eb2ed6c8b64204b4c3a11b41e53ec8 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:38.349748 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30eb2ed6c8b64204b4c3a11b41e53ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.352370 20945 tablet_service.cc:1467] Processing CreateTablet for tablet a9fd7894fc5745ae9588437cd5bd0265 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:38.352788 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fd7894fc5745ae9588437cd5bd0265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.352725 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 4715a768081a49228ab47b8071c40c8f (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:38.353152 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4715a768081a49228ab47b8071c40c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.356141 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 4715a768081a49228ab47b8071c40c8f (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:38.356591 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4715a768081a49228ab47b8071c40c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.360193 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 4715a768081a49228ab47b8071c40c8f (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:38.360600 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4715a768081a49228ab47b8071c40c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.362134 24451 tablet_bootstrap.cc:492] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:38.364182 24452 tablet_bootstrap.cc:492] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:38.364379 24451 tablet_bootstrap.cc:654] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.366473 24452 tablet_bootstrap.cc:654] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.373219 24311 consensus_queue.cc:1035] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.374264 24451 tablet_bootstrap.cc:492] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:38.374418 24451 ts_tablet_manager.cc:1367] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 20:09:38.375206 24451 raft_consensus.cc:348] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.375444 24451 raft_consensus.cc:374] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.375567 24451 raft_consensus.cc:729] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.375808 24451 consensus_queue.cc:260] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.376344 24455 tablet_bootstrap.cc:492] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:38.376652 24344 consensus_queue.cc:1035] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.377960 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:38.379705 24455 tablet_bootstrap.cc:654] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.381232 24344 raft_consensus.cc:482] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:38.381381 24344 raft_consensus.cc:504] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.382741 24344 leader_election.cc:288] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:38.385061 24451 ts_tablet_manager.cc:1398] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 20:09:38.388967 24452 tablet_bootstrap.cc:492] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:38.389145 24452 ts_tablet_manager.cc:1367] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.011s
I20240501 20:09:38.390005 24452 raft_consensus.cc:348] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.390223 24452 raft_consensus.cc:374] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.390331 24452 raft_consensus.cc:729] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.390574 24452 consensus_queue.cc:260] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.391013 24451 tablet_bootstrap.cc:492] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:38.391793 24452 ts_tablet_manager.cc:1398] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 20:09:38.392092 24452 tablet_bootstrap.cc:492] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:38.392823 24451 tablet_bootstrap.cc:654] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.394738 24452 tablet_bootstrap.cc:654] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.397337 24452 tablet_bootstrap.cc:492] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:38.397512 24452 ts_tablet_manager.cc:1367] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:38.398449 24452 raft_consensus.cc:348] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.398711 24452 raft_consensus.cc:374] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.398797 24452 raft_consensus.cc:729] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.399144 24452 consensus_queue.cc:260] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.399950 24452 ts_tablet_manager.cc:1398] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:38.400300 24452 tablet_bootstrap.cc:492] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:38.402089 24452 tablet_bootstrap.cc:654] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.408514 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 8475b2afd7734391a443d4248b5e9145 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbda68b5147e49bc8edce6edffef5276]), partition=
I20240501 20:09:38.408903 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8475b2afd7734391a443d4248b5e9145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.409350 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 8475b2afd7734391a443d4248b5e9145 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbda68b5147e49bc8edce6edffef5276]), partition=
I20240501 20:09:38.409677 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8475b2afd7734391a443d4248b5e9145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.417327 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 8475b2afd7734391a443d4248b5e9145 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbda68b5147e49bc8edce6edffef5276]), partition=
I20240501 20:09:38.417775 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8475b2afd7734391a443d4248b5e9145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.419255 24236 raft_consensus.cc:482] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:38.419394 24236 raft_consensus.cc:504] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.420295 24236 leader_election.cc:288] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:38.421983 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
W20240501 20:09:38.422799 23646 leader_election.cc:341] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:38.439873 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd7894fc5745ae9588437cd5bd0265" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:38.440177 23711 raft_consensus.cc:2454] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:38.440707 23911 leader_election.cc:302] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:38.441020 24311 raft_consensus.cc:2781] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:38.441102 24311 raft_consensus.cc:482] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:38.441227 24311 raft_consensus.cc:3037] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.443240 24464 tablet_bootstrap.cc:492] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:38.444700 24311 raft_consensus.cc:504] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.445708 24464 tablet_bootstrap.cc:654] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.445705 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
W20240501 20:09:38.446348 23644 leader_election.cc:341] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:38.446525 23644 leader_election.cc:302] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46; no voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:38.446790 24236 raft_consensus.cc:2726] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:38.446949 24461 tablet_bootstrap.cc:492] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:38.450306 24461 tablet_bootstrap.cc:654] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.452975 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd7894fc5745ae9588437cd5bd0265" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:38.453265 23711 raft_consensus.cc:3037] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.454399 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd7894fc5745ae9588437cd5bd0265" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:38.455380 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd7894fc5745ae9588437cd5bd0265" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
W20240501 20:09:38.455899 23909 leader_election.cc:341] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:38.456288 23909 leader_election.cc:341] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:38.456585 23711 raft_consensus.cc:2454] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:38.457095 23911 leader_election.cc:302] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters: cda19d6a81c0473db302535ce2ada7f0
I20240501 20:09:38.457492 24344 raft_consensus.cc:2781] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:38.458806 24311 leader_election.cc:288] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:38.469635 24455 tablet_bootstrap.cc:492] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:38.476099 24452 tablet_bootstrap.cc:492] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:38.476261 24452 ts_tablet_manager.cc:1367] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.076s user 0.019s sys 0.029s
I20240501 20:09:38.476387 24461 tablet_bootstrap.cc:492] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:38.476574 24461 ts_tablet_manager.cc:1367] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.004s
I20240501 20:09:38.477092 24452 raft_consensus.cc:348] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:38.477301 24452 raft_consensus.cc:374] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.477382 24452 raft_consensus.cc:729] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.477619 24452 consensus_queue.cc:260] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:38.478225 24452 ts_tablet_manager.cc:1398] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:38.478576 24452 tablet_bootstrap.cc:492] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:38.479051 24461 raft_consensus.cc:348] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.479337 24461 raft_consensus.cc:374] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.479440 24461 raft_consensus.cc:729] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.479777 24461 consensus_queue.cc:260] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.483666 24452 tablet_bootstrap.cc:654] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.484860 24464 tablet_bootstrap.cc:492] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:38.485033 24464 ts_tablet_manager.cc:1367] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.000s
I20240501 20:09:38.490269 24464 raft_consensus.cc:348] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:38.490675 24464 raft_consensus.cc:374] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.490844 24464 raft_consensus.cc:729] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.491299 24464 consensus_queue.cc:260] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:38.492385 24452 tablet_bootstrap.cc:492] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:38.492563 24452 ts_tablet_manager.cc:1367] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240501 20:09:38.493245 24455 ts_tablet_manager.cc:1367] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.118s user 0.000s sys 0.021s
I20240501 20:09:38.493288 24461 ts_tablet_manager.cc:1398] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.017s user 0.002s sys 0.001s
I20240501 20:09:38.493396 24452 raft_consensus.cc:348] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.493608 24452 raft_consensus.cc:374] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.493649 24461 tablet_bootstrap.cc:492] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:38.493676 24452 raft_consensus.cc:729] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.493919 24452 consensus_queue.cc:260] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.492385 24451 tablet_bootstrap.cc:492] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:38.494992 24451 ts_tablet_manager.cc:1367] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.105s user 0.005s sys 0.009s
I20240501 20:09:38.496707 24451 raft_consensus.cc:348] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.497172 24452 ts_tablet_manager.cc:1398] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 20:09:38.494586 24464 ts_tablet_manager.cc:1398] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:09:38.498241 24455 raft_consensus.cc:348] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.499771 23573 raft_consensus.cc:482] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:38.499955 23573 raft_consensus.cc:504] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.500689 23573 leader_election.cc:288] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:38.500742 24451 raft_consensus.cc:374] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.500813 24451 raft_consensus.cc:729] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.501053 24451 consensus_queue.cc:260] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.501493 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd7894fc5745ae9588437cd5bd0265" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:38.501612 24451 ts_tablet_manager.cc:1398] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 20:09:38.501579 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd7894fc5745ae9588437cd5bd0265" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:38.501840 23711 raft_consensus.cc:2379] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cda19d6a81c0473db302535ce2ada7f0 in current term 1: Already voted for candidate 744c5a532e724391830c0785d32eb2fb in this term.
I20240501 20:09:38.501910 24451 tablet_bootstrap.cc:492] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:38.502941 24461 tablet_bootstrap.cc:654] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.503331 24451 tablet_bootstrap.cc:654] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.500618 24344 raft_consensus.cc:686] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:38.503777 24455 raft_consensus.cc:374] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.503890 24455 raft_consensus.cc:729] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.504127 24455 consensus_queue.cc:260] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.504663 24455 ts_tablet_manager.cc:1398] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 20:09:38.504925 24455 tablet_bootstrap.cc:492] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:38.506325 24455 tablet_bootstrap.cc:654] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.509687 24344 consensus_queue.cc:237] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:38.512379 20900 leader_election.cc:302] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:38.512897 23573 raft_consensus.cc:3037] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.516641 20823 catalog_manager.cc:5484] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:38.517534 24455 tablet_bootstrap.cc:492] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:38.517695 24455 ts_tablet_manager.cc:1367] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20240501 20:09:38.517899 23573 raft_consensus.cc:2726] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:38.520454 24455 raft_consensus.cc:348] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.520679 24455 raft_consensus.cc:374] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.520784 24455 raft_consensus.cc:729] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.522413 24455 consensus_queue.cc:260] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.522923 24455 ts_tablet_manager.cc:1398] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:09:38.525470 24236 raft_consensus.cc:482] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:38.525645 24236 raft_consensus.cc:504] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.526360 24236 leader_election.cc:288] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:38.527336 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8475b2afd7734391a443d4248b5e9145" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:38.527567 24110 raft_consensus.cc:2454] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:38.528326 23646 leader_election.cc:302] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:38.528748 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8475b2afd7734391a443d4248b5e9145" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
W20240501 20:09:38.529371 23644 leader_election.cc:341] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:38.529811 24236 raft_consensus.cc:2781] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:38.529918 24236 raft_consensus.cc:482] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:38.529985 24236 raft_consensus.cc:3037] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.532702 24236 raft_consensus.cc:504] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.533624 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8475b2afd7734391a443d4248b5e9145" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:38.533812 24110 raft_consensus.cc:3037] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.535332 24451 tablet_bootstrap.cc:492] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:38.535473 24451 ts_tablet_manager.cc:1367] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20240501 20:09:38.536247 24451 raft_consensus.cc:348] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.536443 24451 raft_consensus.cc:374] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.536538 24451 raft_consensus.cc:729] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.536747 24451 consensus_queue.cc:260] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.537096 24110 raft_consensus.cc:2454] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:38.537302 24451 ts_tablet_manager.cc:1398] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:38.537611 23646 leader_election.cc:302] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:38.538906 24236 leader_election.cc:288] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:38.539259 24236 raft_consensus.cc:2781] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:38.539659 24236 raft_consensus.cc:686] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:38.540419 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8475b2afd7734391a443d4248b5e9145" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:38.540616 23977 raft_consensus.cc:3037] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.543329 23977 raft_consensus.cc:2454] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:38.540133 24236 consensus_queue.cc:237] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.552731 24461 tablet_bootstrap.cc:492] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:38.552886 24461 ts_tablet_manager.cc:1367] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.003s
I20240501 20:09:38.553452 24461 raft_consensus.cc:348] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:38.553614 24461 raft_consensus.cc:374] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.553668 24461 raft_consensus.cc:729] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.553866 24461 consensus_queue.cc:260] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:38.554342 24461 ts_tablet_manager.cc:1398] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:38.567625 20823 catalog_manager.cc:5484] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:38.690546 20822 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 20:09:38.690799 20822 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 20:09:38.693892 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 3fb71fbd8e6d4de3948894bf939aaa65
I20240501 20:09:38.694387 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 71a423af43c04e659a0fa268afe10250
I20240501 20:09:38.694784 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 45083ec2f79e41559827ebce282688c9
I20240501 20:09:38.694742 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb71fbd8e6d4de3948894bf939aaa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:38.694748 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb71fbd8e6d4de3948894bf939aaa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:38.694793 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb71fbd8e6d4de3948894bf939aaa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:38.695044 24201 tablet_replica.cc:330] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:38.695052 24203 tablet_replica.cc:330] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:38.695221 24197 tablet_replica.cc:330] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:38.695278 24201 raft_consensus.cc:2229] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.695281 24203 raft_consensus.cc:2229] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:38.695446 24197 raft_consensus.cc:2229] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.695564 24201 raft_consensus.cc:2258] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.695643 24203 raft_consensus.cc:2258] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.695722 24197 raft_consensus.cc:2258] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.696189 24203 ts_tablet_manager.cc:1875] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.696305 24201 ts_tablet_manager.cc:1875] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.696360 24197 ts_tablet_manager.cc:1875] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.700006 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a423af43c04e659a0fa268afe10250 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:38.700291 24433 tablet_replica.cc:330] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:38.700336 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a423af43c04e659a0fa268afe10250 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:38.700531 24433 raft_consensus.cc:2229] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.700646 24203 ts_tablet_manager.cc:1888] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.700779 24203 log.cc:1196] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/3fb71fbd8e6d4de3948894bf939aaa65
I20240501 20:09:38.700800 24433 raft_consensus.cc:2258] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.700717 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:38.701326 24203 ts_tablet_manager.cc:1909] T 3fb71fbd8e6d4de3948894bf939aaa65 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:38.701566 24433 ts_tablet_manager.cc:1875] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.703450 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 3fb71fbd8e6d4de3948894bf939aaa65 (table rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]) successfully deleted
I20240501 20:09:38.703503 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 45083ec2f79e41559827ebce282688c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:38.703517 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 45083ec2f79e41559827ebce282688c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:38.703848 24203 tablet_replica.cc:330] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:38.704079 24203 raft_consensus.cc:2229] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.704144 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet 45083ec2f79e41559827ebce282688c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:38.704310 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a423af43c04e659a0fa268afe10250 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:38 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:38.704327 24203 raft_consensus.cc:2258] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.705072 24203 ts_tablet_manager.cc:1875] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.706295 24201 ts_tablet_manager.cc:1888] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.706475 24201 log.cc:1196] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/3fb71fbd8e6d4de3948894bf939aaa65
I20240501 20:09:38.706943 24201 ts_tablet_manager.cc:1909] T 3fb71fbd8e6d4de3948894bf939aaa65 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:38.707674 24197 ts_tablet_manager.cc:1888] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.707794 24197 log.cc:1196] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/3fb71fbd8e6d4de3948894bf939aaa65
I20240501 20:09:38.708029 24201 tablet_replica.cc:330] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:38.708268 24197 ts_tablet_manager.cc:1909] T 3fb71fbd8e6d4de3948894bf939aaa65 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:38.708257 24201 raft_consensus.cc:2229] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:38.708657 24201 raft_consensus.cc:2258] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.709336 24197 tablet_replica.cc:330] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:38.709424 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 3fb71fbd8e6d4de3948894bf939aaa65 (table rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]) successfully deleted
I20240501 20:09:38.709561 24197 raft_consensus.cc:2229] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.709966 24201 ts_tablet_manager.cc:1875] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.710652 24197 raft_consensus.cc:2258] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.711275 24203 ts_tablet_manager.cc:1888] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.711370 24197 ts_tablet_manager.cc:1875] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.711416 24203 log.cc:1196] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/45083ec2f79e41559827ebce282688c9
I20240501 20:09:38.711822 24203 ts_tablet_manager.cc:1909] T 45083ec2f79e41559827ebce282688c9 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:38.713799 24433 ts_tablet_manager.cc:1888] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.713987 24433 log.cc:1196] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/71a423af43c04e659a0fa268afe10250
I20240501 20:09:38.714547 24433 ts_tablet_manager.cc:1909] T 71a423af43c04e659a0fa268afe10250 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:38.715330 23977 raft_consensus.cc:1261] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:38.715600 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 45083ec2f79e41559827ebce282688c9 (table rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]) successfully deleted
I20240501 20:09:38.716166 24110 raft_consensus.cc:1261] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:38.716176 24236 consensus_queue.cc:1035] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.716488 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 71a423af43c04e659a0fa268afe10250 (table rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]) successfully deleted
I20240501 20:09:38.716562 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 3fb71fbd8e6d4de3948894bf939aaa65 (table rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]) successfully deleted
I20240501 20:09:38.718364 24236 consensus_queue.cc:1035] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.720767 24201 ts_tablet_manager.cc:1888] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.720853 20820 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet 8475b2afd7734391a443d4248b5e9145 (table rebalancer_dynamic_table [id=bbda68b5147e49bc8edce6edffef5276]). Expected version 1 got 0
I20240501 20:09:38.721038 24201 log.cc:1196] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/45083ec2f79e41559827ebce282688c9
I20240501 20:09:38.721662 24201 ts_tablet_manager.cc:1909] T 45083ec2f79e41559827ebce282688c9 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:38.722497 24480 raft_consensus.cc:482] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:38.722676 24480 raft_consensus.cc:504] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:38.723714 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 45083ec2f79e41559827ebce282688c9 (table rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]) successfully deleted
I20240501 20:09:38.723768 24480 leader_election.cc:288] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:38.724159 24201 tablet_replica.cc:330] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:38.724418 24201 raft_consensus.cc:2229] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:38.724536 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4715a768081a49228ab47b8071c40c8f" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:38.724745 23846 raft_consensus.cc:2454] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:38.724740 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4715a768081a49228ab47b8071c40c8f" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:38.725039 20966 raft_consensus.cc:2454] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:38.725277 23647 leader_election.cc:302] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:38.725881 24480 raft_consensus.cc:2781] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:38.725994 24480 raft_consensus.cc:482] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:38.726078 24480 raft_consensus.cc:3037] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.724771 24201 raft_consensus.cc:2258] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.727727 24201 ts_tablet_manager.cc:1875] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.730448 24480 raft_consensus.cc:504] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:38.730227 24505 tablet.cc:1722] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:38.733420 24505 tablet.cc:1620] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:38.733467 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4715a768081a49228ab47b8071c40c8f" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:38.733657 20966 raft_consensus.cc:3037] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.738654 20822 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet 8475b2afd7734391a443d4248b5e9145 (table rebalancer_dynamic_table [id=bbda68b5147e49bc8edce6edffef5276]). Expected version 1 got 0
I20240501 20:09:38.742799 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4715a768081a49228ab47b8071c40c8f" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:38.743110 23844 raft_consensus.cc:3037] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.743403 24197 ts_tablet_manager.cc:1888] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.743561 24197 log.cc:1196] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/71a423af43c04e659a0fa268afe10250
I20240501 20:09:38.743988 24197 ts_tablet_manager.cc:1909] T 71a423af43c04e659a0fa268afe10250 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:38.744189 20822 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 8475b2afd7734391a443d4248b5e9145 (table rebalancer_dynamic_table [id=bbda68b5147e49bc8edce6edffef5276]). Expected version 1 got 0
I20240501 20:09:38.744741 24201 ts_tablet_manager.cc:1888] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.744935 24201 log.cc:1196] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/71a423af43c04e659a0fa268afe10250
I20240501 20:09:38.745348 24201 ts_tablet_manager.cc:1909] T 71a423af43c04e659a0fa268afe10250 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:38.746762 24197 tablet_replica.cc:330] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:38.746826 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 71a423af43c04e659a0fa268afe10250 (table rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]) successfully deleted
I20240501 20:09:38.746955 24197 raft_consensus.cc:2229] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:38.747229 24197 raft_consensus.cc:2258] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:38.747972 24197 ts_tablet_manager.cc:1875] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:38.747970 23844 raft_consensus.cc:2454] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:38.748605 23647 leader_election.cc:302] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:38.749085 20820 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=bbda68b5147e49bc8edce6edffef5276] alter complete (version 1)
I20240501 20:09:38.750787 20966 raft_consensus.cc:2454] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:38.751000 24480 leader_election.cc:288] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:38.751150 24508 raft_consensus.cc:2781] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:38.751461 24508 raft_consensus.cc:686] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:38.751864 24508 consensus_queue.cc:237] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:38.752538 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 71a423af43c04e659a0fa268afe10250 (table rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]) successfully deleted
I20240501 20:09:38.752962 24504 tablet.cc:1722] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:38.753712 24504 tablet.cc:1620] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:38.754009 24197 ts_tablet_manager.cc:1888] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:38.754139 24197 log.cc:1196] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/45083ec2f79e41559827ebce282688c9
I20240501 20:09:38.754631 24197 ts_tablet_manager.cc:1909] T 45083ec2f79e41559827ebce282688c9 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:38.755672 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 45083ec2f79e41559827ebce282688c9 (table rebalance_test_table_10 [id=1d8009b6b8f74e919b14d28b5a78035c]) successfully deleted
I20240501 20:09:38.755836 24509 tablet.cc:1722] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:38.756471 24509 tablet.cc:1620] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:38.757858 20822 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 8475b2afd7734391a443d4248b5e9145 (table rebalancer_dynamic_table [id=bbda68b5147e49bc8edce6edffef5276]). Expected version 1 got 0
I20240501 20:09:38.760537 20822 catalog_manager.cc:5484] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:38.856065 20822 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:38.858253 24344 raft_consensus.cc:482] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:38.858428 24344 raft_consensus.cc:504] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.859097 24344 leader_election.cc:288] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:38.859826 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:38.860142 23711 raft_consensus.cc:2454] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:38.860280 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:38.860873 24110 raft_consensus.cc:2454] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:38.861538 23911 leader_election.cc:302] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:38.861837 24344 raft_consensus.cc:2781] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:38.861948 24344 raft_consensus.cc:482] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:38.862061 24344 raft_consensus.cc:3037] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.864504 24505 tablet.cc:1722] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:38.864972 24505 tablet.cc:1620] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:38.864894 24504 tablet.cc:1722] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:38.864998 24344 raft_consensus.cc:504] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.865397 24504 tablet.cc:1620] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:38.866503 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:38.866568 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:38.866719 23711 raft_consensus.cc:3037] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.866757 24108 raft_consensus.cc:3037] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.869953 23711 raft_consensus.cc:2454] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:38.870301 24108 raft_consensus.cc:2454] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:38.870659 23911 leader_election.cc:302] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:38.870656 24344 leader_election.cc:288] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:38.870994 24344 raft_consensus.cc:2781] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:38.871146 24344 raft_consensus.cc:686] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:38.871409 24344 consensus_queue.cc:237] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:38.872485 24509 tablet.cc:1722] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:38.873402 24509 tablet.cc:1620] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:38.875624 20820 catalog_manager.cc:5484] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:38.876899 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=bbda68b5147e49bc8edce6edffef5276] alter complete (version 2)
I20240501 20:09:38.913049 21206 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:38.916891 23689 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:38.940992 23956 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:38.963397 24344 consensus_queue.cc:1035] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.975103 23825 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:38.980804 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:38.993460 24504 tablet.cc:1722] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:38.999037 24505 tablet.cc:1722] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:39.004523 24509 tablet.cc:1722] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:39.012681 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=bbda68b5147e49bc8edce6edffef5276] alter complete (version 3)
I20240501 20:09:39.013890 24089 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:39.021629 20945 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:39.047005 24472 consensus_queue.cc:1035] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:39.053759 21076 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:39.112459 21336 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:09:39.121352 20820 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:39.121595 20820 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:39.125595 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 8475b2afd7734391a443d4248b5e9145
I20240501 20:09:39.126579 23956 tablet_service.cc:1514] Processing DeleteTablet for tablet 8475b2afd7734391a443d4248b5e9145 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:39.126821 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 8475b2afd7734391a443d4248b5e9145 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:39.127202 23689 tablet_service.cc:1514] Processing DeleteTablet for tablet 8475b2afd7734391a443d4248b5e9145 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:39.148631 24508 consensus_queue.cc:1035] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:39.170367 24534 tablet_replica.cc:330] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:39.170854 24535 tablet_replica.cc:330] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:39.171517 24535 raft_consensus.cc:2229] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:39.171804 24534 raft_consensus.cc:2229] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.172607 24533 tablet_replica.cc:330] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:39.173995 24534 raft_consensus.cc:2258] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.174170 24535 raft_consensus.cc:2258] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.175982 24507 consensus_queue.cc:1035] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:39.176350 24533 raft_consensus.cc:2229] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.178236 24535 ts_tablet_manager.cc:1875] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.178558 24533 raft_consensus.cc:2258] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.179216 24534 ts_tablet_manager.cc:1875] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.180392 24533 ts_tablet_manager.cc:1875] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.204349 20822 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 20:09:39.204675 20822 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 20:09:39.210429 24534 ts_tablet_manager.cc:1888] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:39.211061 24534 log.cc:1196] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/8475b2afd7734391a443d4248b5e9145
I20240501 20:09:39.212095 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet f832bc442c3a416e83d55eab263ea0ea
I20240501 20:09:39.213647 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet f832bc442c3a416e83d55eab263ea0ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:39.214336 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet f832bc442c3a416e83d55eab263ea0ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:39.214635 24201 tablet_replica.cc:330] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:39.214674 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet f832bc442c3a416e83d55eab263ea0ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:39.214833 24201 raft_consensus.cc:2229] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.214939 24433 tablet_replica.cc:330] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:39.215090 24201 raft_consensus.cc:2258] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.215129 24433 raft_consensus.cc:2229] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:39.215329 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet bd4e9ca0289f44bf881f9e5078f76f4f
I20240501 20:09:39.215453 24433 raft_consensus.cc:2258] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.215835 24201 ts_tablet_manager.cc:1875] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.216121 24433 ts_tablet_manager.cc:1875] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.216176 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 473185b99beb4995a9089fc7c3a42ffe
I20240501 20:09:39.218961 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 473185b99beb4995a9089fc7c3a42ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:39.219928 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4e9ca0289f44bf881f9e5078f76f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:39.220279 24197 tablet_replica.cc:330] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:39.220577 24197 raft_consensus.cc:2229] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.220891 24197 raft_consensus.cc:2258] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.220880 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4e9ca0289f44bf881f9e5078f76f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:39.221521 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4e9ca0289f44bf881f9e5078f76f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:39.221918 24197 ts_tablet_manager.cc:1875] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.222041 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 473185b99beb4995a9089fc7c3a42ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:39.222601 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 473185b99beb4995a9089fc7c3a42ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:39.229985 24433 ts_tablet_manager.cc:1888] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.230265 24433 log.cc:1196] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/f832bc442c3a416e83d55eab263ea0ea
I20240501 20:09:39.231415 24433 ts_tablet_manager.cc:1909] T f832bc442c3a416e83d55eab263ea0ea P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:39.232095 24197 ts_tablet_manager.cc:1888] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.232306 24197 log.cc:1196] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd4e9ca0289f44bf881f9e5078f76f4f
I20240501 20:09:39.232779 24433 tablet_replica.cc:330] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:39.232837 24197 ts_tablet_manager.cc:1909] T bd4e9ca0289f44bf881f9e5078f76f4f P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:39.233078 24433 raft_consensus.cc:2229] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.233167 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet f832bc442c3a416e83d55eab263ea0ea (table rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]) successfully deleted
I20240501 20:09:39.233516 24433 raft_consensus.cc:2258] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.234273 24201 ts_tablet_manager.cc:1888] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.234342 20822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:39.234892 24201 log.cc:1196] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/f832bc442c3a416e83d55eab263ea0ea
I20240501 20:09:39.235531 24433 ts_tablet_manager.cc:1875] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.235515 24201 ts_tablet_manager.cc:1909] T f832bc442c3a416e83d55eab263ea0ea P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:39.237689 24535 ts_tablet_manager.cc:1888] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:39.237823 24535 log.cc:1196] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/8475b2afd7734391a443d4248b5e9145
I20240501 20:09:39.238768 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet f832bc442c3a416e83d55eab263ea0ea (table rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]) successfully deleted
I20240501 20:09:39.239722 24533 ts_tablet_manager.cc:1888] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:39.239859 24533 log.cc:1196] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/8475b2afd7734391a443d4248b5e9145
I20240501 20:09:39.239861 24534 ts_tablet_manager.cc:1909] T 8475b2afd7734391a443d4248b5e9145 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:39.240286 24533 ts_tablet_manager.cc:1909] T 8475b2afd7734391a443d4248b5e9145 P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:39.245196 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 8475b2afd7734391a443d4248b5e9145 (table rebalancer_dynamic_table_new_name [id=bbda68b5147e49bc8edce6edffef5276]) successfully deleted
I20240501 20:09:39.245273 24535 ts_tablet_manager.cc:1909] T 8475b2afd7734391a443d4248b5e9145 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:39.246794 24197 tablet_replica.cc:330] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:39.247009 24197 raft_consensus.cc:2229] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:39.247171 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb421b160a74206968d9261984d934d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ff909ec694c649f1a3aa7ae9cb42eebd]), partition=
I20240501 20:09:39.247282 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb421b160a74206968d9261984d934d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ff909ec694c649f1a3aa7ae9cb42eebd]), partition=
I20240501 20:09:39.247563 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb421b160a74206968d9261984d934d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.247514 24197 raft_consensus.cc:2258] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.247632 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb421b160a74206968d9261984d934d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.248243 24197 ts_tablet_manager.cc:1875] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.251755 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 8475b2afd7734391a443d4248b5e9145 (table rebalancer_dynamic_table_new_name [id=bbda68b5147e49bc8edce6edffef5276]) successfully deleted
I20240501 20:09:39.252034 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 8475b2afd7734391a443d4248b5e9145 (table rebalancer_dynamic_table_new_name [id=bbda68b5147e49bc8edce6edffef5276]) successfully deleted
I20240501 20:09:39.253929 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet bd4e9ca0289f44bf881f9e5078f76f4f (table rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]) successfully deleted
I20240501 20:09:39.254338 24197 ts_tablet_manager.cc:1888] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.254494 24197 log.cc:1196] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/473185b99beb4995a9089fc7c3a42ffe
I20240501 20:09:39.254935 24197 ts_tablet_manager.cc:1909] T 473185b99beb4995a9089fc7c3a42ffe P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:39.256134 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb421b160a74206968d9261984d934d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ff909ec694c649f1a3aa7ae9cb42eebd]), partition=
I20240501 20:09:39.256218 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 473185b99beb4995a9089fc7c3a42ffe (table rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]) successfully deleted
I20240501 20:09:39.256539 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb421b160a74206968d9261984d934d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.261018 24201 tablet_replica.cc:330] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:39.261281 24201 raft_consensus.cc:2229] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.261520 24201 raft_consensus.cc:2258] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.261559 24540 tablet_bootstrap.cc:492] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:39.262251 24201 ts_tablet_manager.cc:1875] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.263948 24540 tablet_bootstrap.cc:654] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.269583 24433 ts_tablet_manager.cc:1888] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.269729 24433 log.cc:1196] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/473185b99beb4995a9089fc7c3a42ffe
I20240501 20:09:39.270166 24540 tablet_bootstrap.cc:492] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:39.270459 24539 tablet_replica.cc:330] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:39.270680 24539 raft_consensus.cc:2229] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.270817 24542 tablet_bootstrap.cc:492] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:39.270988 24539 raft_consensus.cc:2258] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.271056 24540 ts_tablet_manager.cc:1367] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 20:09:39.271663 24433 ts_tablet_manager.cc:1909] T 473185b99beb4995a9089fc7c3a42ffe P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:39.271775 24539 ts_tablet_manager.cc:1875] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.272918 24542 tablet_bootstrap.cc:654] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.273977 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 473185b99beb4995a9089fc7c3a42ffe (table rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]) successfully deleted
I20240501 20:09:39.274227 24541 tablet_bootstrap.cc:492] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:39.276218 24541 tablet_bootstrap.cc:654] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.276485 24201 ts_tablet_manager.cc:1888] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.276670 24540 raft_consensus.cc:348] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.277264 24201 log.cc:1196] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd4e9ca0289f44bf881f9e5078f76f4f
I20240501 20:09:39.277791 24201 ts_tablet_manager.cc:1909] T bd4e9ca0289f44bf881f9e5078f76f4f P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
W20240501 20:09:39.278595 21031 consensus_peers.cc:485] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:39.278944 24540 raft_consensus.cc:374] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.278995 24539 ts_tablet_manager.cc:1888] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.279122 24540 raft_consensus.cc:729] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.279250 24539 log.cc:1196] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/f832bc442c3a416e83d55eab263ea0ea
I20240501 20:09:39.279407 24540 consensus_queue.cc:260] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.279808 24539 ts_tablet_manager.cc:1909] T f832bc442c3a416e83d55eab263ea0ea P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:39.281904 24539 tablet_replica.cc:330] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:39.282325 24539 raft_consensus.cc:2229] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:39.282632 24539 raft_consensus.cc:2258] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.282804 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet f832bc442c3a416e83d55eab263ea0ea (table rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]) successfully deleted
I20240501 20:09:39.283411 24539 ts_tablet_manager.cc:1875] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.285190 24201 tablet_replica.cc:330] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:39.285394 24201 raft_consensus.cc:2229] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.285672 24201 raft_consensus.cc:2258] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.285677 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet bd4e9ca0289f44bf881f9e5078f76f4f (table rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]) successfully deleted
I20240501 20:09:39.286397 24201 ts_tablet_manager.cc:1875] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.290843 24539 ts_tablet_manager.cc:1888] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.291071 24539 log.cc:1196] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/bd4e9ca0289f44bf881f9e5078f76f4f
I20240501 20:09:39.291303 24540 ts_tablet_manager.cc:1398] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240501 20:09:39.291697 24539 ts_tablet_manager.cc:1909] T bd4e9ca0289f44bf881f9e5078f76f4f P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:39.292665 24542 tablet_bootstrap.cc:492] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:39.292886 24542 ts_tablet_manager.cc:1367] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20240501 20:09:39.293807 24542 raft_consensus.cc:348] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.294083 24542 raft_consensus.cc:374] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.294235 24542 raft_consensus.cc:729] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.294559 24542 consensus_queue.cc:260] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.294641 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet bd4e9ca0289f44bf881f9e5078f76f4f (table rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]) successfully deleted
I20240501 20:09:39.295213 24542 ts_tablet_manager.cc:1398] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:39.302176 24201 ts_tablet_manager.cc:1888] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.302309 24201 log.cc:1196] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/473185b99beb4995a9089fc7c3a42ffe
I20240501 20:09:39.302714 24201 ts_tablet_manager.cc:1909] T 473185b99beb4995a9089fc7c3a42ffe P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:39.304272 24541 tablet_bootstrap.cc:492] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:39.304489 24541 ts_tablet_manager.cc:1367] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.011s
I20240501 20:09:39.304875 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 473185b99beb4995a9089fc7c3a42ffe (table rebalance_test_table_12 [id=c3a8af06da964a46ac847c202ba3c8f5]) successfully deleted
I20240501 20:09:39.305567 24541 raft_consensus.cc:348] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.305954 24541 raft_consensus.cc:374] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.306111 24541 raft_consensus.cc:729] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.306480 24541 consensus_queue.cc:260] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.307819 24541 ts_tablet_manager.cc:1398] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:39.370698 21358 consensus_queue.cc:237] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
W20240501 20:09:39.373448 21293 consensus_peers.cc:485] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aebb2a23076449fd9db492ed3e3b0cbf. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:39.373447 21228 raft_consensus.cc:1261] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:39.374287 20966 raft_consensus.cc:1261] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:39.374143 24565 consensus_queue.cc:1035] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:39.375133 24565 consensus_queue.cc:1035] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:39.378432 24171 raft_consensus.cc:2932] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:39.379052 20966 raft_consensus.cc:2932] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:39.382624 21228 raft_consensus.cc:2932] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:39.384119 20822 catalog_manager.cc:5484] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:39.457188 24344 consensus_queue.cc:1035] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:39.459982 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:39.475200 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 68133e033c824e11a1ed744bff76a934 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.475716 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68133e033c824e11a1ed744bff76a934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.476719 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 6b52c4f7c7e848139f2e115618fbd402 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.476907 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 68133e033c824e11a1ed744bff76a934 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.477108 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b52c4f7c7e848139f2e115618fbd402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.477329 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68133e033c824e11a1ed744bff76a934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.479281 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 6b52c4f7c7e848139f2e115618fbd402 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.481833 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b52c4f7c7e848139f2e115618fbd402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.481770 24344 consensus_queue.cc:1035] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:39.485157 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 6b52c4f7c7e848139f2e115618fbd402 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.485576 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b52c4f7c7e848139f2e115618fbd402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.489660 24542 tablet_bootstrap.cc:492] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:39.491084 24090 tablet_service.cc:1467] Processing CreateTablet for tablet 68133e033c824e11a1ed744bff76a934 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.491281 24542 tablet_bootstrap.cc:654] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.491483 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68133e033c824e11a1ed744bff76a934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.492105 24590 tablet_bootstrap.cc:492] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:39.493808 24590 tablet_bootstrap.cc:654] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.496450 23824 tablet_service.cc:1467] Processing CreateTablet for tablet ed53558d93214d34aedca8505870ec61 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.496868 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed53558d93214d34aedca8505870ec61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.499065 24088 tablet_service.cc:1467] Processing CreateTablet for tablet ed53558d93214d34aedca8505870ec61 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.499589 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed53558d93214d34aedca8505870ec61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.501382 21208 tablet_service.cc:1467] Processing CreateTablet for tablet ed53558d93214d34aedca8505870ec61 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.501807 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed53558d93214d34aedca8505870ec61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.505836 24590 tablet_bootstrap.cc:492] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:39.506228 24540 tablet_bootstrap.cc:492] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:39.507836 24540 tablet_bootstrap.cc:654] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.506063 24590 ts_tablet_manager.cc:1367] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240501 20:09:39.511989 24590 raft_consensus.cc:348] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:39.513172 24590 raft_consensus.cc:374] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.513414 24590 raft_consensus.cc:729] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.513788 24590 consensus_queue.cc:260] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:39.514788 24590 ts_tablet_manager.cc:1398] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:39.518177 24217 raft_consensus.cc:482] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.518352 24217 raft_consensus.cc:504] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.519148 24217 leader_election.cc:288] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:39.519412 24541 tablet_bootstrap.cc:492] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:39.520128 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb421b160a74206968d9261984d934d" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:39.520354 23844 raft_consensus.cc:2454] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:39.522154 24541 tablet_bootstrap.cc:654] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.523468 24046 leader_election.cc:302] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:39.523833 24542 tablet_bootstrap.cc:492] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:39.524008 24542 ts_tablet_manager.cc:1367] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.002s
I20240501 20:09:39.524526 24217 raft_consensus.cc:2781] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:39.524638 24217 raft_consensus.cc:482] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:39.524727 24217 raft_consensus.cc:3037] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.524688 24542 raft_consensus.cc:348] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:39.524890 24542 raft_consensus.cc:374] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.524958 24542 raft_consensus.cc:729] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.525205 24542 consensus_queue.cc:260] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:39.526072 24542 ts_tablet_manager.cc:1398] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:39.527793 24217 raft_consensus.cc:504] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.529222 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb421b160a74206968d9261984d934d" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:39.529457 23844 raft_consensus.cc:3037] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.534813 24507 raft_consensus.cc:482] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.535017 24507 raft_consensus.cc:504] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:39.535796 24507 leader_election.cc:288] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:39.537065 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b52c4f7c7e848139f2e115618fbd402" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:09:39.537122 23646 leader_election.cc:341] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Not found: Tablet not found: 6b52c4f7c7e848139f2e115618fbd402
W20240501 20:09:39.537847 23647 leader_election.cc:341] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:39.537976 23647 leader_election.cc:302] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46; no voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:39.540616 24507 raft_consensus.cc:2726] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:39.542222 23844 raft_consensus.cc:2454] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:39.542912 24046 leader_election.cc:302] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:39.553171 24540 tablet_bootstrap.cc:492] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:39.553457 24540 ts_tablet_manager.cc:1367] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.021s
I20240501 20:09:39.554349 24217 leader_election.cc:288] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:39.554579 24217 raft_consensus.cc:2781] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:39.554723 24217 raft_consensus.cc:686] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:39.554559 24540 raft_consensus.cc:348] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:39.555019 24540 raft_consensus.cc:374] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.555147 24540 raft_consensus.cc:729] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.555109 24217 consensus_queue.cc:237] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.555610 24540 consensus_queue.cc:260] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:39.557987 24540 ts_tablet_manager.cc:1398] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:39.558436 24540 tablet_bootstrap.cc:492] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:39.562346 24540 tablet_bootstrap.cc:654] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.571750 24540 tablet_bootstrap.cc:492] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:39.572031 24540 ts_tablet_manager.cc:1367] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240501 20:09:39.573017 24540 raft_consensus.cc:348] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:39.573361 24540 raft_consensus.cc:374] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.573508 24540 raft_consensus.cc:729] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.573843 24540 consensus_queue.cc:260] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:39.575392 24540 ts_tablet_manager.cc:1398] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:39.578127 24600 tablet_bootstrap.cc:492] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:39.583478 24600 tablet_bootstrap.cc:654] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.598326 24598 tablet_bootstrap.cc:492] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:39.600849 24598 tablet_bootstrap.cc:654] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
W20240501 20:09:39.604673 24046 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 61783.3us
I20240501 20:09:39.605993 24541 tablet_bootstrap.cc:492] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:39.606086 24192 raft_consensus.cc:482] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.606182 24541 ts_tablet_manager.cc:1367] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.000s
I20240501 20:09:39.606253 24192 raft_consensus.cc:504] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:39.607143 24192 leader_election.cc:288] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:39.607192 24541 raft_consensus.cc:348] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:39.607481 24541 raft_consensus.cc:374] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.607611 24541 raft_consensus.cc:729] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.607918 24541 consensus_queue.cc:260] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:39.608584 24541 ts_tablet_manager.cc:1398] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:39.608444 20820 catalog_manager.cc:5484] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:39.608917 24541 tablet_bootstrap.cc:492] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:39.613149 24541 tablet_bootstrap.cc:654] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.632221 24598 tablet_bootstrap.cc:492] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:39.632535 24598 ts_tablet_manager.cc:1367] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240501 20:09:39.633631 24598 raft_consensus.cc:348] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:39.634157 24598 raft_consensus.cc:374] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.634320 24598 raft_consensus.cc:729] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.634793 24598 consensus_queue.cc:260] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:39.636751 24598 ts_tablet_manager.cc:1398] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:39.651934 24600 tablet_bootstrap.cc:492] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:39.655782 24600 ts_tablet_manager.cc:1367] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.078s user 0.001s sys 0.010s
I20240501 20:09:39.661398 24600 raft_consensus.cc:348] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:39.662374 24600 raft_consensus.cc:374] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.662815 24600 raft_consensus.cc:729] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.665452 24600 consensus_queue.cc:260] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:39.666668 24600 ts_tablet_manager.cc:1398] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:09:39.672291 24541 tablet_bootstrap.cc:492] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:39.672500 24541 ts_tablet_manager.cc:1367] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.064s user 0.013s sys 0.016s
I20240501 20:09:39.673784 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb421b160a74206968d9261984d934d" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:39.674067 20966 raft_consensus.cc:2454] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:39.674033 24541 raft_consensus.cc:348] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:39.674230 24541 raft_consensus.cc:374] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.674321 24541 raft_consensus.cc:729] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.674557 24541 consensus_queue.cc:260] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:39.675098 24541 ts_tablet_manager.cc:1398] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:39.675097 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb421b160a74206968d9261984d934d" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:39.675328 20966 raft_consensus.cc:3037] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.675380 24541 tablet_bootstrap.cc:492] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:39.677218 24541 tablet_bootstrap.cc:654] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.678447 20966 raft_consensus.cc:2454] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:39.684376 24541 tablet_bootstrap.cc:492] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:39.684546 24541 ts_tablet_manager.cc:1367] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 20:09:39.685441 24541 raft_consensus.cc:348] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:39.685674 24541 raft_consensus.cc:374] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.685822 24541 raft_consensus.cc:729] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.686442 24541 consensus_queue.cc:260] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:39.687129 24541 ts_tablet_manager.cc:1398] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:39.692755 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb421b160a74206968d9261984d934d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:39.693123 23844 raft_consensus.cc:2379] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cda19d6a81c0473db302535ce2ada7f0 in current term 1: Already voted for candidate 8559229e044b4e1ea661b881b3f9af50 in this term.
I20240501 20:09:39.696928 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb421b160a74206968d9261984d934d" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:39.697700 20902 leader_election.cc:302] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0; no voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:39.698261 24192 raft_consensus.cc:2726] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:39.718720 20820 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 20:09:39.718979 20820 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 20:09:39.724084 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 07a4fb0b08244e47ab330bafe40991bd
I20240501 20:09:39.724762 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet b5a05a1a27f54527b4d9bfa05faf5434
I20240501 20:09:39.725219 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a4fb0b08244e47ab330bafe40991bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:39.725234 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a4fb0b08244e47ab330bafe40991bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:39.725399 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a05a1a27f54527b4d9bfa05faf5434 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:39.725641 24539 tablet_replica.cc:330] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:39.725814 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 26c37bbdd8d74983a4966478aec924fe
I20240501 20:09:39.725852 24539 raft_consensus.cc:2229] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:39.725989 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a05a1a27f54527b4d9bfa05faf5434 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:39.726222 24539 raft_consensus.cc:2258] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.725989 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a4fb0b08244e47ab330bafe40991bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:39.726974 24539 ts_tablet_manager.cc:1875] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.727209 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c37bbdd8d74983a4966478aec924fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:39.730757 24197 tablet_replica.cc:330] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:39.730981 24197 raft_consensus.cc:2229] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.731205 24197 raft_consensus.cc:2258] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.731752 24201 tablet_replica.cc:330] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:39.731885 24197 ts_tablet_manager.cc:1875] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.731983 24201 raft_consensus.cc:2229] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.732182 24201 raft_consensus.cc:2258] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.732860 24201 ts_tablet_manager.cc:1875] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.732975 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a05a1a27f54527b4d9bfa05faf5434 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:39.735724 24433 tablet_replica.cc:330] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:39.735967 24433 raft_consensus.cc:2229] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.736253 24433 raft_consensus.cc:2258] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.736974 24433 ts_tablet_manager.cc:1875] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.740916 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c37bbdd8d74983a4966478aec924fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:39.742098 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c37bbdd8d74983a4966478aec924fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:39 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:39.745465 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:39.751981 20966 raft_consensus.cc:1261] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:39.753922 24217 consensus_queue.cc:1035] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:39.754375 24201 ts_tablet_manager.cc:1888] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.754519 24201 log.cc:1196] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/b5a05a1a27f54527b4d9bfa05faf5434
I20240501 20:09:39.755071 24201 ts_tablet_manager.cc:1909] T b5a05a1a27f54527b4d9bfa05faf5434 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:39.757154 24433 ts_tablet_manager.cc:1888] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.757305 24433 log.cc:1196] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/b5a05a1a27f54527b4d9bfa05faf5434
I20240501 20:09:39.757526 23844 raft_consensus.cc:1261] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:09:39.758154 21162 consensus_peers.cc:485] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:39.758512 24539 ts_tablet_manager.cc:1888] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.758775 24539 log.cc:1196] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/07a4fb0b08244e47ab330bafe40991bd
I20240501 20:09:39.759135 24217 consensus_queue.cc:1035] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:39.759380 24539 ts_tablet_manager.cc:1909] T 07a4fb0b08244e47ab330bafe40991bd P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:39.761917 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 07a4fb0b08244e47ab330bafe40991bd (table rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]) successfully deleted
I20240501 20:09:39.762210 24433 ts_tablet_manager.cc:1909] T b5a05a1a27f54527b4d9bfa05faf5434 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:39.763554 24539 tablet_replica.cc:330] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:39.763928 24539 raft_consensus.cc:2229] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.764739 24539 raft_consensus.cc:2258] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.767935 24539 ts_tablet_manager.cc:1875] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.778079 24197 ts_tablet_manager.cc:1888] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.778296 24197 log.cc:1196] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/07a4fb0b08244e47ab330bafe40991bd
I20240501 20:09:39.778784 24197 ts_tablet_manager.cc:1909] T 07a4fb0b08244e47ab330bafe40991bd P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:39.778750 24593 mvcc.cc:205] Tried to move back new op lower bound from 7022977760258371584 to 7022977759460642816. Current Snapshot: MvccSnapshot[applied={T|T < 7022977760258371584}]
I20240501 20:09:39.779951 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 07a4fb0b08244e47ab330bafe40991bd (table rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]) successfully deleted
I20240501 20:09:39.780798 24633 tablet.cc:1722] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:39.781627 24629 mvcc.cc:205] Tried to move back new op lower bound from 7022977760258371584 to 7022977759460642816. Current Snapshot: MvccSnapshot[applied={T|T < 7022977760258371584}]
I20240501 20:09:39.781764 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet b5a05a1a27f54527b4d9bfa05faf5434 (table rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]) successfully deleted
I20240501 20:09:39.782711 24633 tablet.cc:1620] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:39.785585 24201 tablet_replica.cc:330] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:39.786470 24201 raft_consensus.cc:2229] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.786665 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet b5a05a1a27f54527b4d9bfa05faf5434 (table rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]) successfully deleted
I20240501 20:09:39.787230 24201 raft_consensus.cc:2258] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.789258 24201 ts_tablet_manager.cc:1875] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.789717 20820 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 3fb421b160a74206968d9261984d934d (table rebalancer_dynamic_table [id=ff909ec694c649f1a3aa7ae9cb42eebd]). Expected version 1 got 0
I20240501 20:09:39.788100 24197 tablet_replica.cc:330] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:39.790097 24197 raft_consensus.cc:2229] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:39.792047 24632 tablet.cc:1722] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:39.792214 24197 raft_consensus.cc:2258] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.792914 24197 ts_tablet_manager.cc:1875] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.793129 24632 tablet.cc:1620] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:39.790745 24192 raft_consensus.cc:482] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.793974 24192 raft_consensus.cc:504] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:39.794701 24192 leader_election.cc:288] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:39.795430 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68133e033c824e11a1ed744bff76a934" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:39.795660 24110 raft_consensus.cc:2454] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:39.796164 20902 leader_election.cc:302] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:39.796706 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=ff909ec694c649f1a3aa7ae9cb42eebd] alter complete (version 1)
I20240501 20:09:39.796980 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68133e033c824e11a1ed744bff76a934" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:39.797231 23977 raft_consensus.cc:2454] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:39.797436 24539 ts_tablet_manager.cc:1888] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.797605 24539 log.cc:1196] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/26c37bbdd8d74983a4966478aec924fe
I20240501 20:09:39.797996 24192 raft_consensus.cc:2781] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:39.798091 24539 ts_tablet_manager.cc:1909] T 26c37bbdd8d74983a4966478aec924fe P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:39.798112 24192 raft_consensus.cc:482] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:39.798209 24192 raft_consensus.cc:3037] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.799146 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 26c37bbdd8d74983a4966478aec924fe (table rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]) successfully deleted
I20240501 20:09:39.800582 24192 raft_consensus.cc:504] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:39.801716 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68133e033c824e11a1ed744bff76a934" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:39.801925 23977 raft_consensus.cc:3037] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.804234 23977 raft_consensus.cc:2454] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:39.804888 20900 leader_election.cc:302] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:39.805186 23573 raft_consensus.cc:2781] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:39.805752 23573 raft_consensus.cc:686] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:39.806152 23573 consensus_queue.cc:237] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:39.807719 24192 leader_election.cc:288] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:39.808184 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68133e033c824e11a1ed744bff76a934" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:39.808393 24110 raft_consensus.cc:3037] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.811167 24110 raft_consensus.cc:2454] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:39.815419 24630 ts_tablet_manager.cc:902] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:39.825950 24201 ts_tablet_manager.cc:1888] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.826141 24201 log.cc:1196] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/07a4fb0b08244e47ab330bafe40991bd
I20240501 20:09:39.826756 24201 ts_tablet_manager.cc:1909] T 07a4fb0b08244e47ab330bafe40991bd P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:39.827286 24201 tablet_replica.cc:330] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:39.827643 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 07a4fb0b08244e47ab330bafe40991bd (table rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]) successfully deleted
I20240501 20:09:39.838640 20822 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 3fb421b160a74206968d9261984d934d (table rebalancer_dynamic_table [id=ff909ec694c649f1a3aa7ae9cb42eebd]). Expected version 1 got 0
I20240501 20:09:39.838948 20822 catalog_manager.cc:5484] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:39.853785 24648 tablet.cc:1722] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:39.853988 23689 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:39.854528 24648 tablet.cc:1620] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:39.856303 24197 ts_tablet_manager.cc:1888] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.856448 24197 log.cc:1196] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/b5a05a1a27f54527b4d9bfa05faf5434
I20240501 20:09:39.856894 24197 ts_tablet_manager.cc:1909] T b5a05a1a27f54527b4d9bfa05faf5434 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:39.857973 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet b5a05a1a27f54527b4d9bfa05faf5434 (table rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]) successfully deleted
I20240501 20:09:39.827512 24201 raft_consensus.cc:2229] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:39.858497 24201 raft_consensus.cc:2258] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.858568 20822 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 3fb421b160a74206968d9261984d934d (table rebalancer_dynamic_table [id=ff909ec694c649f1a3aa7ae9cb42eebd]). Expected version 1 got 0
I20240501 20:09:39.859164 24201 ts_tablet_manager.cc:1875] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.863773 24630 tablet_copy_client.cc:308] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:09:39.864193 24197 tablet_replica.cc:330] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:39.864444 24197 raft_consensus.cc:2229] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.874790 24197 raft_consensus.cc:2258] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.876767 24197 ts_tablet_manager.cc:1875] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:39.877755 21206 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:39.887998 24088 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:39.893043 24201 ts_tablet_manager.cc:1888] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.897778 24201 log.cc:1196] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/26c37bbdd8d74983a4966478aec924fe
I20240501 20:09:39.898496 24201 ts_tablet_manager.cc:1909] T 26c37bbdd8d74983a4966478aec924fe P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:39.899168 21336 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:09:39.900681 24197 ts_tablet_manager.cc:1888] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:39.900978 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 26c37bbdd8d74983a4966478aec924fe (table rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]) successfully deleted
I20240501 20:09:39.901065 24197 log.cc:1196] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/26c37bbdd8d74983a4966478aec924fe
I20240501 20:09:39.901818 24197 ts_tablet_manager.cc:1909] T 26c37bbdd8d74983a4966478aec924fe P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:39.905797 20822 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:39.906126 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 26c37bbdd8d74983a4966478aec924fe (table rebalance_test_table_14 [id=b6968900e9de4cd38300c3a79afc60e4]) successfully deleted
I20240501 20:09:39.916241 24632 tablet.cc:1722] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:39.916749 24632 tablet.cc:1620] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:39.919611 24633 tablet.cc:1722] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:39.920095 24633 tablet.cc:1620] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:39.933759 20820 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=ff909ec694c649f1a3aa7ae9cb42eebd] alter complete (version 2)
I20240501 20:09:39.940797 24313 raft_consensus.cc:482] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.940955 24313 raft_consensus.cc:504] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:39.941764 24313 leader_election.cc:288] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:39.944837 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:39.946779 23956 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:39.951220 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet aebb2a23076449fd9db492ed3e3b0cbf from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:37097)
I20240501 20:09:39.951561 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet aebb2a23076449fd9db492ed3e3b0cbf from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:37097: session id = 51bf640e895b4e5eb91bdf7509ea7e46-aebb2a23076449fd9db492ed3e3b0cbf
I20240501 20:09:39.956430 21378 tablet_copy_source_session.cc:205] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:39.957355 24313 raft_consensus.cc:482] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.957528 24313 raft_consensus.cc:504] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:39.958191 24313 leader_election.cc:288] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:39.958938 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b52c4f7c7e848139f2e115618fbd402" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:39.959168 23711 raft_consensus.cc:2454] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:39.960516 23781 leader_election.cc:302] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:39.960844 24110 raft_consensus.cc:2454] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:39.961004 24630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebb2a23076449fd9db492ed3e3b0cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.961311 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b52c4f7c7e848139f2e115618fbd402" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:39.961879 23781 leader_election.cc:302] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:39.962342 24313 raft_consensus.cc:2781] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:39.962426 24313 raft_consensus.cc:482] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:39.962509 24313 raft_consensus.cc:3037] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.967868 24630 tablet_copy_client.cc:783] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:39.968209 24630 tablet_copy_client.cc:647] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:39.972608 20945 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:39.975840 24313 raft_consensus.cc:504] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:39.976560 24313 leader_election.cc:288] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:39.976754 24313 raft_consensus.cc:2781] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:39.976845 24313 raft_consensus.cc:482] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:39.979102 24630 tablet_copy_client.cc:523] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:39.979943 24313 raft_consensus.cc:3037] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.982946 21076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:39.987061 24108 raft_consensus.cc:2454] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:39.989187 24313 raft_consensus.cc:504] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:39.989851 24313 leader_election.cc:288] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
W20240501 20:09:40.000402 24043 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 84706.5us
I20240501 20:09:40.003224 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b52c4f7c7e848139f2e115618fbd402" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:40.003469 24110 raft_consensus.cc:3037] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.006347 24110 raft_consensus.cc:2454] T 6b52c4f7c7e848139f2e115618fbd402 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:40.007306 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b52c4f7c7e848139f2e115618fbd402" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:40.007596 23711 raft_consensus.cc:3037] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.010524 23711 raft_consensus.cc:2454] T 6b52c4f7c7e848139f2e115618fbd402 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:40.012032 24648 tablet.cc:1722] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:40.012550 24648 tablet.cc:1620] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:40.014508 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:40.014761 24110 raft_consensus.cc:3037] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.017621 24110 raft_consensus.cc:2454] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:40.018453 23781 leader_election.cc:302] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:40.018887 23781 leader_election.cc:302] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:40.019460 24313 raft_consensus.cc:2781] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:40.019865 24660 raft_consensus.cc:2781] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:40.022106 23824 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:40.028071 24630 tablet_bootstrap.cc:492] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:40.043783 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:40.069317 24632 tablet.cc:1722] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:40.074262 24633 tablet.cc:1722] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:40.079773 20820 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=ff909ec694c649f1a3aa7ae9cb42eebd] alter complete (version 3)
I20240501 20:09:40.081280 24648 tablet.cc:1722] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:40.084867 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:40.085145 21226 raft_consensus.cc:3037] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.087952 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:40.088430 24660 raft_consensus.cc:686] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:40.088747 24660 consensus_queue.cc:237] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:40.089336 21226 raft_consensus.cc:2454] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:40.089761 24313 raft_consensus.cc:686] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:40.090029 24313 consensus_queue.cc:237] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.116201 20820 catalog_manager.cc:5484] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:40.122349 20820 catalog_manager.cc:5484] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:40.175729 24630 tablet_bootstrap.cc:492] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:40.176265 24630 tablet_bootstrap.cc:492] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:09:40.176623 24630 ts_tablet_manager.cc:1367] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.149s user 0.048s sys 0.006s
I20240501 20:09:40.177517 24630 raft_consensus.cc:348] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.177767 24630 raft_consensus.cc:729] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.178098 24630 consensus_queue.cc:260] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.186571 20820 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:40.186758 20820 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:40.190591 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 3fb421b160a74206968d9261984d934d
I20240501 20:09:40.191468 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb421b160a74206968d9261984d934d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:40.192101 24668 tablet_replica.cc:330] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:40.192332 24668 raft_consensus.cc:2229] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:40.192939 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb421b160a74206968d9261984d934d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:40.193002 24668 raft_consensus.cc:2258] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.193312 24201 tablet_replica.cc:330] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:40.193572 24201 raft_consensus.cc:2229] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.193769 24668 ts_tablet_manager.cc:1875] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.193825 24201 raft_consensus.cc:2258] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.195282 24201 ts_tablet_manager.cc:1875] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.198940 24668 ts_tablet_manager.cc:1888] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:40.199082 24668 log.cc:1196] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/3fb421b160a74206968d9261984d934d
I20240501 20:09:40.199589 24668 ts_tablet_manager.cc:1909] T 3fb421b160a74206968d9261984d934d P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:40.200650 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 3fb421b160a74206968d9261984d934d (table rebalancer_dynamic_table_new_name [id=ff909ec694c649f1a3aa7ae9cb42eebd]) successfully deleted
I20240501 20:09:40.204110 24630 ts_tablet_manager.cc:1398] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.027s user 0.021s sys 0.003s
I20240501 20:09:40.205087 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-aebb2a23076449fd9db492ed3e3b0cbf received from {username='slave'} at 127.13.251.133:37097
I20240501 20:09:40.205197 24201 ts_tablet_manager.cc:1888] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:40.205272 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-aebb2a23076449fd9db492ed3e3b0cbf on tablet aebb2a23076449fd9db492ed3e3b0cbf with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:09:40.205394 24201 log.cc:1196] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/3fb421b160a74206968d9261984d934d
I20240501 20:09:40.205902 24201 ts_tablet_manager.cc:1909] T 3fb421b160a74206968d9261984d934d P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:40.207489 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 3fb421b160a74206968d9261984d934d (table rebalancer_dynamic_table_new_name [id=ff909ec694c649f1a3aa7ae9cb42eebd]) successfully deleted
I20240501 20:09:40.211877 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb421b160a74206968d9261984d934d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:40.212850 24669 tablet_replica.cc:330] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:40.213335 24669 raft_consensus.cc:2229] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.213647 24669 raft_consensus.cc:2258] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.215108 24669 ts_tablet_manager.cc:1875] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.217458 24192 consensus_queue.cc:1035] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.223717 24634 consensus_queue.cc:1035] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:09:40.228927 20822 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 20:09:40.229565 20822 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 20:09:40.232585 23711 raft_consensus.cc:1203] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:40.236366 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 99fb3a1b19924865b1c10fa129aa9192
I20240501 20:09:40.237135 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 99fb3a1b19924865b1c10fa129aa9192 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:40.237187 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 591bdca6da8849f4a6300b0db04d3754
I20240501 20:09:40.237228 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 99fb3a1b19924865b1c10fa129aa9192 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:40.237421 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 99fb3a1b19924865b1c10fa129aa9192 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:40.237684 24201 tablet_replica.cc:330] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:40.237854 24539 tablet_replica.cc:330] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:40.237907 24201 raft_consensus.cc:2229] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:40.238042 24539 raft_consensus.cc:2229] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.238224 24201 raft_consensus.cc:2258] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.238314 24539 raft_consensus.cc:2258] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.238962 24201 ts_tablet_manager.cc:1875] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.238951 24539 ts_tablet_manager.cc:1875] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.243701 24433 tablet_replica.cc:330] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:40.243716 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 591bdca6da8849f4a6300b0db04d3754 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:40.243924 24433 raft_consensus.cc:2229] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.244014 24197 tablet_replica.cc:330] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:40.244108 24669 ts_tablet_manager.cc:1888] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:40.244205 24433 raft_consensus.cc:2258] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.244258 24669 log.cc:1196] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/3fb421b160a74206968d9261984d934d
I20240501 20:09:40.244258 24197 raft_consensus.cc:2229] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.244527 24197 raft_consensus.cc:2258] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.244935 24433 ts_tablet_manager.cc:1875] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.244899 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet d61793ef4b84426c839c55e2fabfe1f6
I20240501 20:09:40.245208 24197 ts_tablet_manager.cc:1875] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.246019 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet d61793ef4b84426c839c55e2fabfe1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:40.247342 24669 ts_tablet_manager.cc:1909] T 3fb421b160a74206968d9261984d934d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:09:40.248546 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet d61793ef4b84426c839c55e2fabfe1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:40.248633 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 591bdca6da8849f4a6300b0db04d3754 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:40.248975 24539 ts_tablet_manager.cc:1888] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.249095 24539 log.cc:1196] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/99fb3a1b19924865b1c10fa129aa9192
I20240501 20:09:40.249549 24539 ts_tablet_manager.cc:1909] T 99fb3a1b19924865b1c10fa129aa9192 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:40.249637 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet d61793ef4b84426c839c55e2fabfe1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:40.249652 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 591bdca6da8849f4a6300b0db04d3754 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:40.250496 24539 tablet_replica.cc:330] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:40.250644 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 99fb3a1b19924865b1c10fa129aa9192 (table rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]) successfully deleted
I20240501 20:09:40.250793 24539 raft_consensus.cc:2229] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.251070 24539 raft_consensus.cc:2258] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.251359 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 3fb421b160a74206968d9261984d934d (table rebalancer_dynamic_table_new_name [id=ff909ec694c649f1a3aa7ae9cb42eebd]) successfully deleted
I20240501 20:09:40.252146 24539 ts_tablet_manager.cc:1875] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.252839 24197 ts_tablet_manager.cc:1888] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.253047 24197 log.cc:1196] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/591bdca6da8849f4a6300b0db04d3754
I20240501 20:09:40.253523 24197 ts_tablet_manager.cc:1909] T 591bdca6da8849f4a6300b0db04d3754 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:40.254017 24433 ts_tablet_manager.cc:1888] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.254489 24197 tablet_replica.cc:330] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:40.254709 24197 raft_consensus.cc:2229] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:40.254809 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 591bdca6da8849f4a6300b0db04d3754 (table rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]) successfully deleted
I20240501 20:09:40.255043 24197 raft_consensus.cc:2258] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.255708 24197 ts_tablet_manager.cc:1875] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.258685 24201 ts_tablet_manager.cc:1888] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.258901 24433 log.cc:1196] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/99fb3a1b19924865b1c10fa129aa9192
I20240501 20:09:40.261011 24433 ts_tablet_manager.cc:1909] T 99fb3a1b19924865b1c10fa129aa9192 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:40.261245 24201 log.cc:1196] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/99fb3a1b19924865b1c10fa129aa9192
I20240501 20:09:40.261732 24201 ts_tablet_manager.cc:1909] T 99fb3a1b19924865b1c10fa129aa9192 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:40.261953 24197 ts_tablet_manager.cc:1888] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.262156 24197 log.cc:1196] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/d61793ef4b84426c839c55e2fabfe1f6
I20240501 20:09:40.262627 24197 ts_tablet_manager.cc:1909] T d61793ef4b84426c839c55e2fabfe1f6 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:40.262758 24433 tablet_replica.cc:330] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:40.262845 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 99fb3a1b19924865b1c10fa129aa9192 (table rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]) successfully deleted
I20240501 20:09:40.262971 24433 raft_consensus.cc:2229] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.263221 24433 raft_consensus.cc:2258] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.263844 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet d61793ef4b84426c839c55e2fabfe1f6 (table rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]) successfully deleted
I20240501 20:09:40.264607 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 99fb3a1b19924865b1c10fa129aa9192 (table rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]) successfully deleted
I20240501 20:09:40.264775 24539 ts_tablet_manager.cc:1888] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.264956 24539 log.cc:1196] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/591bdca6da8849f4a6300b0db04d3754
I20240501 20:09:40.264933 24201 tablet_replica.cc:330] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:40.265239 24201 raft_consensus.cc:2229] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.265383 24539 ts_tablet_manager.cc:1909] T 591bdca6da8849f4a6300b0db04d3754 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:40.265545 24201 raft_consensus.cc:2258] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.266330 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 591bdca6da8849f4a6300b0db04d3754 (table rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]) successfully deleted
I20240501 20:09:40.266677 24433 ts_tablet_manager.cc:1875] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.267729 24201 ts_tablet_manager.cc:1875] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.273897 24201 ts_tablet_manager.cc:1888] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.274031 24201 log.cc:1196] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/d61793ef4b84426c839c55e2fabfe1f6
I20240501 20:09:40.274463 24201 ts_tablet_manager.cc:1909] T d61793ef4b84426c839c55e2fabfe1f6 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:40.274801 24433 ts_tablet_manager.cc:1888] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.274972 24433 log.cc:1196] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/d61793ef4b84426c839c55e2fabfe1f6
I20240501 20:09:40.275430 24433 ts_tablet_manager.cc:1909] T d61793ef4b84426c839c55e2fabfe1f6 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:40.275615 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet d61793ef4b84426c839c55e2fabfe1f6 (table rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]) successfully deleted
I20240501 20:09:40.276543 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet d61793ef4b84426c839c55e2fabfe1f6 (table rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]) successfully deleted
I20240501 20:09:40.276800 24201 tablet_replica.cc:330] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:40.277030 24201 raft_consensus.cc:2229] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:40.277434 24201 raft_consensus.cc:2258] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.278256 24201 ts_tablet_manager.cc:1875] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.283967 24201 ts_tablet_manager.cc:1888] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.284119 24201 log.cc:1196] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/591bdca6da8849f4a6300b0db04d3754
I20240501 20:09:40.284572 24201 ts_tablet_manager.cc:1909] T 591bdca6da8849f4a6300b0db04d3754 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:40.285553 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 591bdca6da8849f4a6300b0db04d3754 (table rebalance_test_table_16 [id=74e6041608884b28936597b5b5f1040e]) successfully deleted
I20240501 20:09:40.294446 20822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:40.302181 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 16ab6a45179644e79032c298d309f79a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24e4f27ebbb445bf8990672e3a971fef]), partition=
I20240501 20:09:40.302580 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ab6a45179644e79032c298d309f79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.302862 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 16ab6a45179644e79032c298d309f79a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24e4f27ebbb445bf8990672e3a971fef]), partition=
I20240501 20:09:40.302918 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 16ab6a45179644e79032c298d309f79a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24e4f27ebbb445bf8990672e3a971fef]), partition=
I20240501 20:09:40.303225 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ab6a45179644e79032c298d309f79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.303309 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ab6a45179644e79032c298d309f79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.315402 24686 tablet_bootstrap.cc:492] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:40.318276 24686 tablet_bootstrap.cc:654] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.318393 24685 tablet_bootstrap.cc:492] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:40.319967 24687 tablet_bootstrap.cc:492] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:40.322639 24687 tablet_bootstrap.cc:654] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.324023 24685 tablet_bootstrap.cc:654] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.324198 24686 tablet_bootstrap.cc:492] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:40.324406 24686 ts_tablet_manager.cc:1367] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 20:09:40.325368 24686 raft_consensus.cc:348] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.325768 24686 raft_consensus.cc:374] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.325992 24686 raft_consensus.cc:729] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.326404 24686 consensus_queue.cc:260] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.327275 24686 ts_tablet_manager.cc:1398] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:40.334331 24685 tablet_bootstrap.cc:492] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:40.334638 24685 ts_tablet_manager.cc:1367] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240501 20:09:40.336019 24685 raft_consensus.cc:348] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.336273 24687 tablet_bootstrap.cc:492] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:40.336315 24685 raft_consensus.cc:374] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.336445 24685 raft_consensus.cc:729] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.336464 24687 ts_tablet_manager.cc:1367] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240501 20:09:40.336781 24685 consensus_queue.cc:260] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.337266 24687 raft_consensus.cc:348] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.337432 24685 ts_tablet_manager.cc:1398] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:40.337560 24687 raft_consensus.cc:374] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.337669 24687 raft_consensus.cc:729] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.338389 24687 consensus_queue.cc:260] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.339835 24687 ts_tablet_manager.cc:1398] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:40.347491 21357 consensus_queue.cc:237] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:40.349967 21226 raft_consensus.cc:1261] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:40.350677 24171 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.352381 20965 raft_consensus.cc:1261] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:40.352910 24171 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.356124 24565 raft_consensus.cc:2932] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:40.356791 20965 raft_consensus.cc:2932] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:40.361871 21226 raft_consensus.cc:2932] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:40.363037 20825 catalog_manager.cc:5484] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
W20240501 20:09:40.364444 21291 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Couldn't send request to peer 744c5a532e724391830c0785d32eb2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30d2aada073b4d8989d7db8be4034756. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:40.398260 21226 consensus_queue.cc:237] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:40.401196 21097 raft_consensus.cc:1261] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:40.401136 21357 raft_consensus.cc:1261] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:40.402082 24546 consensus_queue.cc:1035] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.403471 24354 consensus_queue.cc:1035] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.418579 24354 raft_consensus.cc:2932] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:40.419346 21357 raft_consensus.cc:2932] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:40.423471 21097 raft_consensus.cc:2932] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:40.425565 20825 catalog_manager.cc:5484] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
W20240501 20:09:40.428056 21163 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1e295a46a174f3f8673e03f57890b58. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:40.465759 23977 consensus_queue.cc:237] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:40.468607 23844 raft_consensus.cc:1261] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:40.468665 24110 raft_consensus.cc:1261] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:40.469285 24666 consensus_queue.cc:1035] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:40.469770 24666 consensus_queue.cc:1035] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:09:40.471565 23909 consensus_peers.cc:485] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ded048b4bb94eb79f95554db7e15b7b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:40.474584 24660 consensus_queue.cc:1035] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.493680 24660 consensus_queue.cc:1035] T 6b52c4f7c7e848139f2e115618fbd402 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.504026 24660 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.510152 24313 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.534813 24472 raft_consensus.cc:2932] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:40.535575 23844 raft_consensus.cc:2932] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:40.540455 24110 raft_consensus.cc:2932] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:40.554256 20825 catalog_manager.cc:5484] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:40.607982 20965 consensus_queue.cc:237] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.610556 21097 raft_consensus.cc:1261] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:40.610549 21226 raft_consensus.cc:1261] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:09:40.610596 20902 consensus_peers.cc:485] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a396d90619784ba0ab865e6d07237db7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:40.611078 23573 consensus_queue.cc:1035] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.611337 24634 consensus_queue.cc:1035] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.614416 24745 raft_consensus.cc:482] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:40.614564 24745 raft_consensus.cc:504] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.615274 24745 leader_election.cc:288] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:40.615828 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ab6a45179644e79032c298d309f79a" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:40.615939 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ab6a45179644e79032c298d309f79a" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:40.616075 24110 raft_consensus.cc:2454] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:40.616154 23712 raft_consensus.cc:2454] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:40.616530 23911 leader_election.cc:302] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:40.616806 24745 raft_consensus.cc:2781] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:40.616920 24745 raft_consensus.cc:482] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:40.617015 24745 raft_consensus.cc:3037] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.618826 24745 raft_consensus.cc:504] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.619326 24745 leader_election.cc:288] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:40.619772 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ab6a45179644e79032c298d309f79a" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:40.619947 23712 raft_consensus.cc:3037] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.621835 23712 raft_consensus.cc:2454] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:40.622256 23911 leader_election.cc:302] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:40.622241 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ab6a45179644e79032c298d309f79a" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:40.622439 24110 raft_consensus.cc:3037] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.622609 24745 raft_consensus.cc:2781] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:40.622891 24745 raft_consensus.cc:686] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:40.623220 24745 consensus_queue.cc:237] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.625236 24110 raft_consensus.cc:2454] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:40.626879 20822 catalog_manager.cc:5484] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:40.627030 24634 raft_consensus.cc:2932] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:40.627789 21097 raft_consensus.cc:2932] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:40.633044 20822 catalog_manager.cc:5484] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:40.634497 21226 raft_consensus.cc:2932] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:40.673941 21097 consensus_queue.cc:237] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:40.678100 21226 raft_consensus.cc:1261] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:40.678200 21357 raft_consensus.cc:1261] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:40.678843 24783 consensus_queue.cc:1035] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.681275 24028 consensus_queue.cc:1035] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.684036 24028 raft_consensus.cc:2932] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
W20240501 20:09:40.684944 21031 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Couldn't send request to peer 744c5a532e724391830c0785d32eb2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a115fe66d11143ea8e24aa891e4e2f33. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:40.688483 21357 raft_consensus.cc:2932] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:40.688848 21226 raft_consensus.cc:2932] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:40.689899 20825 catalog_manager.cc:5484] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:40.726526 21226 consensus_queue.cc:237] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:40.733212 21097 raft_consensus.cc:1261] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:40.733330 21357 raft_consensus.cc:1261] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:40.734040 20822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:09:40.735841 21163 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43e80df9af0e40c5acc4dea95d7e95a8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:40.736369 24546 consensus_queue.cc:1035] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.736721 24715 consensus_queue.cc:1035] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:40.741441 24546 raft_consensus.cc:2932] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:40.743813 23689 tablet_service.cc:1467] Processing CreateTablet for tablet a1cbbb090cb34465b6dac9c35cdba0dc (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.744087 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 30ea740ef5534b089c77063a8d8b9aaa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.744220 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cbbb090cb34465b6dac9c35cdba0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.744513 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ea740ef5534b089c77063a8d8b9aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.746068 21357 raft_consensus.cc:2932] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:40.748996 20820 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 20:09:40.749080 23824 tablet_service.cc:1467] Processing CreateTablet for tablet a1cbbb090cb34465b6dac9c35cdba0dc (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.749234 20820 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 20:09:40.749579 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cbbb090cb34465b6dac9c35cdba0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.750416 24686 tablet_bootstrap.cc:492] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:40.751943 24686 tablet_bootstrap.cc:654] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.751906 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 30ea740ef5534b089c77063a8d8b9aaa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.752300 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ea740ef5534b089c77063a8d8b9aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.752554 20825 catalog_manager.cc:5484] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:40.757194 21076 tablet_service.cc:1467] Processing CreateTablet for tablet a1cbbb090cb34465b6dac9c35cdba0dc (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.759189 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 09a3d74fea894fb6b0e7fc58257ca56a
I20240501 20:09:40.759882 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2
I20240501 20:09:40.760465 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a3d74fea894fb6b0e7fc58257ca56a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:40.760691 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:40.760771 24201 tablet_replica.cc:330] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:40.760934 24433 tablet_replica.cc:330] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:40.760985 24201 raft_consensus.cc:2229] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.761159 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 4 replicas of tablet a396d90619784ba0ab865e6d07237db7
I20240501 20:09:40.761142 24433 raft_consensus.cc:2229] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:40.761313 24201 raft_consensus.cc:2258] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.761508 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:40.762059 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet a396d90619784ba0ab865e6d07237db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:40.762055 24201 ts_tablet_manager.cc:1875] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.762339 23689 tablet_service.cc:1514] Processing DeleteTablet for tablet a396d90619784ba0ab865e6d07237db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:40.762393 24433 raft_consensus.cc:2258] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.763100 24433 ts_tablet_manager.cc:1875] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.767179 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a3d74fea894fb6b0e7fc58257ca56a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:40.767951 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:40.768186 24810 tablet_replica.cc:330] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:40.768267 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet a396d90619784ba0ab865e6d07237db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:40.768421 24810 raft_consensus.cc:2229] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.769852 24810 raft_consensus.cc:2258] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.770881 24810 ts_tablet_manager.cc:1875] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.770963 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a3d74fea894fb6b0e7fc58257ca56a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:40.771319 21078 tablet_service.cc:1514] Processing DeleteTablet for tablet a396d90619784ba0ab865e6d07237db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:40 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:40.771842 24201 ts_tablet_manager.cc:1888] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.772053 24201 log.cc:1196] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/09a3d74fea894fb6b0e7fc58257ca56a
I20240501 20:09:40.772591 24201 ts_tablet_manager.cc:1909] T 09a3d74fea894fb6b0e7fc58257ca56a P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:40.773591 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 09a3d74fea894fb6b0e7fc58257ca56a (table rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]) successfully deleted
I20240501 20:09:40.773641 24201 tablet_replica.cc:330] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:40.773834 24809 tablet_bootstrap.cc:492] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:40.773955 24201 raft_consensus.cc:2229] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.774279 24201 raft_consensus.cc:2258] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.775477 24201 ts_tablet_manager.cc:1875] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.776628 24809 tablet_bootstrap.cc:654] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.782227 24201 ts_tablet_manager.cc:1888] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.783182 24810 ts_tablet_manager.cc:1888] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.786279 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 30ea740ef5534b089c77063a8d8b9aaa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.786871 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ea740ef5534b089c77063a8d8b9aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.787618 24810 log.cc:1196] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/09a3d74fea894fb6b0e7fc58257ca56a
I20240501 20:09:40.788206 24810 ts_tablet_manager.cc:1909] T 09a3d74fea894fb6b0e7fc58257ca56a P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:40.788894 24201 log.cc:1196] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/a3fd6fe82bda4d14b3efe2fc2ec0ddc2
I20240501 20:09:40.789333 24201 ts_tablet_manager.cc:1909] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:40.789364 24810 tablet_replica.cc:330] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:40.789577 24810 raft_consensus.cc:2229] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.789764 24810 raft_consensus.cc:2258] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.791808 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2 (table rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]) successfully deleted
I20240501 20:09:40.791956 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 09a3d74fea894fb6b0e7fc58257ca56a (table rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]) successfully deleted
I20240501 20:09:40.792580 24201 tablet_replica.cc:330] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:40.793058 24201 raft_consensus.cc:2229] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:40.794010 24201 raft_consensus.cc:2258] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.796185 24810 ts_tablet_manager.cc:1875] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.796365 24201 ts_tablet_manager.cc:1875] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.798182 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2e66020ffb438fad69c2ea73762aa7 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.798624 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2e66020ffb438fad69c2ea73762aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.799218 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2e66020ffb438fad69c2ea73762aa7 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.799770 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2e66020ffb438fad69c2ea73762aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.804507 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2e66020ffb438fad69c2ea73762aa7 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.804919 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cbbb090cb34465b6dac9c35cdba0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.804934 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2e66020ffb438fad69c2ea73762aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.804886 21097 raft_consensus.cc:2932] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:40.806470 24201 ts_tablet_manager.cc:1888] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:40.806584 24201 log.cc:1196] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/a396d90619784ba0ab865e6d07237db7
I20240501 20:09:40.807014 24201 ts_tablet_manager.cc:1909] T a396d90619784ba0ab865e6d07237db7 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:40.812383 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet a396d90619784ba0ab865e6d07237db7 (table rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]) successfully deleted
I20240501 20:09:40.815253 24687 tablet_bootstrap.cc:492] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:40.817533 24687 tablet_bootstrap.cc:654] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.821732 24433 ts_tablet_manager.cc:1888] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.822018 24433 log.cc:1196] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/a3fd6fe82bda4d14b3efe2fc2ec0ddc2
I20240501 20:09:40.829823 24685 tablet_bootstrap.cc:492] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:40.831940 24685 tablet_bootstrap.cc:654] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.833201 24433 ts_tablet_manager.cc:1909] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:40.838169 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2 (table rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]) successfully deleted
I20240501 20:09:40.841915 24811 tablet_replica.cc:330] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:40.842214 24811 raft_consensus.cc:2229] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:40.842442 24687 tablet_bootstrap.cc:492] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:40.842653 24687 ts_tablet_manager.cc:1367] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20240501 20:09:40.842729 24811 raft_consensus.cc:2258] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.843758 24811 ts_tablet_manager.cc:1875] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.843600 24687 raft_consensus.cc:348] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:40.843902 24687 raft_consensus.cc:374] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.843969 24687 raft_consensus.cc:729] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.847360 20822 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:40.846032 24687 consensus_queue.cc:260] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:40.852389 24687 ts_tablet_manager.cc:1398] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:09:40.855016 24685 tablet_bootstrap.cc:492] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:40.855260 24685 ts_tablet_manager.cc:1367] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20240501 20:09:40.855590 24687 tablet_bootstrap.cc:492] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:40.856206 24685 raft_consensus.cc:348] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:40.856549 24685 raft_consensus.cc:374] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.856678 24685 raft_consensus.cc:729] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.857059 24685 consensus_queue.cc:260] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:40.857901 24685 ts_tablet_manager.cc:1398] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:40.860581 24814 tablet_bootstrap.cc:492] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:40.864389 24687 tablet_bootstrap.cc:654] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.867040 24814 tablet_bootstrap.cc:654] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.870615 24686 tablet_bootstrap.cc:492] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:40.871127 24814 tablet_bootstrap.cc:492] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:40.871222 24686 ts_tablet_manager.cc:1367] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.121s user 0.031s sys 0.026s
I20240501 20:09:40.871959 24814 ts_tablet_manager.cc:1367] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 20:09:40.873023 24814 raft_consensus.cc:348] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.873370 24810 ts_tablet_manager.cc:1888] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.872517 24686 raft_consensus.cc:348] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.873517 24686 raft_consensus.cc:374] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.873903 24686 raft_consensus.cc:729] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.874415 24814 raft_consensus.cc:374] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.874526 24814 raft_consensus.cc:729] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.874477 24686 consensus_queue.cc:260] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.875087 24814 consensus_queue.cc:260] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.876097 24810 log.cc:1196] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/a3fd6fe82bda4d14b3efe2fc2ec0ddc2
I20240501 20:09:40.876892 24686 ts_tablet_manager.cc:1398] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:09:40.877671 24686 tablet_bootstrap.cc:492] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
W20240501 20:09:40.878381 20812 catalog_manager.cc:4797] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): delete failed for tablet a396d90619784ba0ab865e6d07237db7 because the tablet was not found. No further retry: Not found: Tablet not found: a396d90619784ba0ab865e6d07237db7
I20240501 20:09:40.878475 24809 tablet_bootstrap.cc:492] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:40.878715 24809 ts_tablet_manager.cc:1367] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.105s user 0.040s sys 0.028s
I20240501 20:09:40.879746 24809 raft_consensus.cc:348] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:40.880038 24809 raft_consensus.cc:374] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.880272 24809 raft_consensus.cc:729] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.881268 24686 tablet_bootstrap.cc:654] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.882720 24809 consensus_queue.cc:260] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:40.882926 24810 ts_tablet_manager.cc:1909] T a3fd6fe82bda4d14b3efe2fc2ec0ddc2 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:40.883577 24809 ts_tablet_manager.cc:1398] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:40.884064 24809 tablet_bootstrap.cc:492] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:40.885651 24809 tablet_bootstrap.cc:654] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.877079 24814 ts_tablet_manager.cc:1398] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:40.886094 24817 ts_tablet_manager.cc:902] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:40.886373 24810 tablet_replica.cc:330] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:40.886670 24810 raft_consensus.cc:2229] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.887070 24810 raft_consensus.cc:2258] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.887238 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet a3fd6fe82bda4d14b3efe2fc2ec0ddc2 (table rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]) successfully deleted
I20240501 20:09:40.889186 24810 ts_tablet_manager.cc:1875] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.896673 23712 raft_consensus.cc:1261] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:40.897483 24745 consensus_queue.cc:1035] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.899112 24810 ts_tablet_manager.cc:1888] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:40.908303 24687 tablet_bootstrap.cc:492] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:40.908485 24687 ts_tablet_manager.cc:1367] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.000s
I20240501 20:09:40.909272 24687 raft_consensus.cc:348] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:40.909488 24687 raft_consensus.cc:374] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.899684 24810 log.cc:1196] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/a396d90619784ba0ab865e6d07237db7
I20240501 20:09:40.909585 24687 raft_consensus.cc:729] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.909830 24687 consensus_queue.cc:260] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:40.910421 24687 ts_tablet_manager.cc:1398] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:40.912741 24110 raft_consensus.cc:1261] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:40.914996 24745 consensus_queue.cc:1035] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.915848 24810 ts_tablet_manager.cc:1909] T a396d90619784ba0ab865e6d07237db7 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:40.921293 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet a396d90619784ba0ab865e6d07237db7 (table rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]) successfully deleted
I20240501 20:09:40.925011 24217 raft_consensus.cc:482] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:40.925246 24217 raft_consensus.cc:504] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:40.926087 24217 leader_election.cc:288] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:40.927592 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2e66020ffb438fad69c2ea73762aa7" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:09:40.929621 24046 leader_election.cc:341] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:40.938194 20825 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 16ab6a45179644e79032c298d309f79a (table rebalancer_dynamic_table [id=24e4f27ebbb445bf8990672e3a971fef]). Expected version 1 got 0
I20240501 20:09:40.949266 24811 ts_tablet_manager.cc:1888] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:40.950145 24811 log.cc:1196] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/09a3d74fea894fb6b0e7fc58257ca56a
I20240501 20:09:40.950753 24809 tablet_bootstrap.cc:492] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:40.950767 24811 ts_tablet_manager.cc:1909] T 09a3d74fea894fb6b0e7fc58257ca56a P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:40.950973 24809 ts_tablet_manager.cc:1367] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.067s user 0.026s sys 0.025s
I20240501 20:09:40.951964 24809 raft_consensus.cc:348] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.952204 24809 raft_consensus.cc:374] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.952296 24809 raft_consensus.cc:729] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.952638 24809 consensus_queue.cc:260] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:40.953646 24811 tablet_replica.cc:330] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:40.953791 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 09a3d74fea894fb6b0e7fc58257ca56a (table rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]) successfully deleted
I20240501 20:09:40.954169 24811 raft_consensus.cc:2229] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:40.954761 24811 raft_consensus.cc:2258] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:40.956290 24811 ts_tablet_manager.cc:1875] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:40.956418 24809 ts_tablet_manager.cc:1398] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:40.958907 24809 tablet_bootstrap.cc:492] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:40.961485 20825 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 16ab6a45179644e79032c298d309f79a (table rebalancer_dynamic_table [id=24e4f27ebbb445bf8990672e3a971fef]). Expected version 1 got 0
I20240501 20:09:40.963456 24809 tablet_bootstrap.cc:654] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.988471 24809 tablet_bootstrap.cc:492] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:40.988655 24809 ts_tablet_manager.cc:1367] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.014s
I20240501 20:09:40.989490 24809 raft_consensus.cc:348] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:40.989708 24809 raft_consensus.cc:374] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.989802 24809 raft_consensus.cc:729] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.990337 24811 ts_tablet_manager.cc:1888] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:40.990499 24811 log.cc:1196] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/a396d90619784ba0ab865e6d07237db7
I20240501 20:09:40.991088 24811 ts_tablet_manager.cc:1909] T a396d90619784ba0ab865e6d07237db7 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:40.992224 24809 consensus_queue.cc:260] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:40.992338 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet a396d90619784ba0ab865e6d07237db7 (table rebalance_test_table_18 [id=0219c0f154564ae7ad517432db3c0a1b]) successfully deleted
I20240501 20:09:40.992805 24809 ts_tablet_manager.cc:1398] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:40.993964 24686 tablet_bootstrap.cc:492] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:40.994146 24686 ts_tablet_manager.cc:1367] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.117s user 0.015s sys 0.004s
I20240501 20:09:40.995152 24686 raft_consensus.cc:348] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:40.995409 24686 raft_consensus.cc:374] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.000042 24832 tablet.cc:1722] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:41.000690 24832 tablet.cc:1620] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:41.001610 24837 tablet.cc:1722] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:41.002166 24837 tablet.cc:1620] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:41.008584 20825 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 16ab6a45179644e79032c298d309f79a (table rebalancer_dynamic_table [id=24e4f27ebbb445bf8990672e3a971fef]). Expected version 1 got 0
I20240501 20:09:40.995496 24686 raft_consensus.cc:729] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.009696 24686 consensus_queue.cc:260] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:41.010919 24839 ts_tablet_manager.cc:902] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:41.011855 24686 ts_tablet_manager.cc:1398] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240501 20:09:41.014171 24830 ts_tablet_manager.cc:902] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:41.018946 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=24e4f27ebbb445bf8990672e3a971fef] alter complete (version 1)
I20240501 20:09:41.020159 24817 tablet_copy_client.cc:308] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:09:41.025133 24843 tablet.cc:1722] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:41.025811 24843 tablet.cc:1620] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:41.027343 20825 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 16ab6a45179644e79032c298d309f79a (table rebalancer_dynamic_table [id=24e4f27ebbb445bf8990672e3a971fef]). Expected version 1 got 0
I20240501 20:09:41.034323 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2e66020ffb438fad69c2ea73762aa7" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:41.034648 23712 raft_consensus.cc:2454] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:41.035619 24045 leader_election.cc:302] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters: c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:41.036151 24217 raft_consensus.cc:2781] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.036255 24217 raft_consensus.cc:482] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.036378 24217 raft_consensus.cc:3037] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.040237 24217 raft_consensus.cc:504] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:41.041424 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2e66020ffb438fad69c2ea73762aa7" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:41.041687 23844 raft_consensus.cc:3037] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.044617 23844 raft_consensus.cc:2454] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:41.045127 24046 leader_election.cc:302] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:41.048256 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2e66020ffb438fad69c2ea73762aa7" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:41.048471 23712 raft_consensus.cc:3037] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.050977 24217 leader_election.cc:288] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:41.051646 24217 raft_consensus.cc:2781] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.056376 24217 raft_consensus.cc:686] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:41.056931 24217 consensus_queue.cc:237] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:41.058506 23712 raft_consensus.cc:2454] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:41.071583 24839 tablet_copy_client.cc:308] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:09:41.079852 24830 tablet_copy_client.cc:308] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:09:41.081049 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet 2ded048b4bb94eb79f95554db7e15b7b from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:58569)
I20240501 20:09:41.081144 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet c1e295a46a174f3f8673e03f57890b58 from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:42501)
I20240501 20:09:41.081285 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet 2ded048b4bb94eb79f95554db7e15b7b from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:58569: session id = cda19d6a81c0473db302535ce2ada7f0-2ded048b4bb94eb79f95554db7e15b7b
I20240501 20:09:41.081385 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet c1e295a46a174f3f8673e03f57890b58 from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:42501: session id = 8559229e044b4e1ea661b881b3f9af50-c1e295a46a174f3f8673e03f57890b58
I20240501 20:09:41.085114 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 30d2aada073b4d8989d7db8be4034756 from peer 744c5a532e724391830c0785d32eb2fb ({username='slave'} at 127.13.251.135:46111)
I20240501 20:09:41.085279 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 30d2aada073b4d8989d7db8be4034756 from peer 744c5a532e724391830c0785d32eb2fb at {username='slave'} at 127.13.251.135:46111: session id = 744c5a532e724391830c0785d32eb2fb-30d2aada073b4d8989d7db8be4034756
I20240501 20:09:41.087209 20825 catalog_manager.cc:5484] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:41.087669 21378 tablet_copy_source_session.cc:205] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:41.087666 23997 tablet_copy_source_session.cc:205] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:41.084985 21248 tablet_copy_source_session.cc:205] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:41.091305 24817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d2aada073b4d8989d7db8be4034756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.095746 24839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e295a46a174f3f8673e03f57890b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.099123 24830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ded048b4bb94eb79f95554db7e15b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.102860 24817 tablet_copy_client.cc:783] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:41.103168 24839 tablet_copy_client.cc:783] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:41.103277 24817 tablet_copy_client.cc:647] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:41.103646 24839 tablet_copy_client.cc:647] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:41.106335 24839 tablet_copy_client.cc:523] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:41.109346 24817 tablet_copy_client.cc:523] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:41.110363 24839 tablet_bootstrap.cc:492] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:41.113250 24217 raft_consensus.cc:482] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.113452 24217 raft_consensus.cc:504] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:41.114660 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ea740ef5534b089c77063a8d8b9aaa" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:41.114816 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ea740ef5534b089c77063a8d8b9aaa" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:41.114904 23977 raft_consensus.cc:2454] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:41.115029 23844 raft_consensus.cc:2454] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:41.115576 24043 leader_election.cc:302] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:41.115914 24848 raft_consensus.cc:2781] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.116043 24848 raft_consensus.cc:482] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.116200 24848 raft_consensus.cc:3037] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.118525 24217 leader_election.cc:288] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:41.121714 24848 raft_consensus.cc:504] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:41.122901 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ea740ef5534b089c77063a8d8b9aaa" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:41.123121 23844 raft_consensus.cc:3037] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.125097 24848 leader_election.cc:288] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:41.125671 24817 tablet_bootstrap.cc:492] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:41.125792 23844 raft_consensus.cc:2454] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:41.126847 24046 leader_election.cc:302] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:41.127101 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ea740ef5534b089c77063a8d8b9aaa" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:41.127346 23977 raft_consensus.cc:3037] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.127460 24848 raft_consensus.cc:2781] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.127704 24848 raft_consensus.cc:686] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:41.128065 24848 consensus_queue.cc:237] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:41.131664 24830 tablet_copy_client.cc:783] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:41.132036 24830 tablet_copy_client.cc:647] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:41.132570 23977 raft_consensus.cc:2454] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:41.140545 24830 tablet_copy_client.cc:523] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:41.142038 20825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:41.144548 24830 tablet_bootstrap.cc:492] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:41.153895 24832 tablet.cc:1722] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:41.154387 24832 tablet.cc:1620] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:41.154832 24837 tablet.cc:1722] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:41.155324 24837 tablet.cc:1620] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:41.157032 24843 tablet.cc:1722] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:41.157482 24843 tablet.cc:1620] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:41.169840 24858 ts_tablet_manager.cc:902] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:41.171685 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=24e4f27ebbb445bf8990672e3a971fef] alter complete (version 2)
I20240501 20:09:41.180258 24787 raft_consensus.cc:482] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.180472 24787 raft_consensus.cc:504] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:41.181546 24787 leader_election.cc:288] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:41.183634 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cbbb090cb34465b6dac9c35cdba0dc" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:41.183867 23844 raft_consensus.cc:2454] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:41.184604 21034 leader_election.cc:302] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:41.185146 24787 raft_consensus.cc:2781] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.185273 24787 raft_consensus.cc:482] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.185355 24787 raft_consensus.cc:3037] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.185528 20825 catalog_manager.cc:5484] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:41.188410 24787 raft_consensus.cc:504] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:41.189438 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cbbb090cb34465b6dac9c35cdba0dc" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:41.189659 23844 raft_consensus.cc:3037] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.192175 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cbbb090cb34465b6dac9c35cdba0dc" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:41.192468 23712 raft_consensus.cc:2454] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:41.192590 23844 raft_consensus.cc:2454] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:41.192978 21034 leader_election.cc:302] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:41.193307 24783 raft_consensus.cc:2781] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.193737 24787 leader_election.cc:288] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:41.193898 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cbbb090cb34465b6dac9c35cdba0dc" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:41.194114 23712 raft_consensus.cc:3037] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.196812 23712 raft_consensus.cc:2454] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:41.200042 24830 tablet_bootstrap.cc:492] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:41.200645 24830 tablet_bootstrap.cc:492] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:09:41.201001 24830 ts_tablet_manager.cc:1367] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.057s user 0.019s sys 0.000s
I20240501 20:09:41.201989 24830 raft_consensus.cc:348] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:41.202351 24830 raft_consensus.cc:729] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.202697 24830 consensus_queue.cc:260] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:41.203579 24830 ts_tablet_manager.cc:1398] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:41.204769 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-2ded048b4bb94eb79f95554db7e15b7b received from {username='slave'} at 127.13.251.129:58569
I20240501 20:09:41.204973 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-2ded048b4bb94eb79f95554db7e15b7b on tablet 2ded048b4bb94eb79f95554db7e15b7b with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:09:41.213086 24783 raft_consensus.cc:686] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:41.213513 24783 consensus_queue.cc:237] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:41.221617 24858 tablet_copy_client.cc:308] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:09:41.222889 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet a115fe66d11143ea8e24aa891e4e2f33 from peer 744c5a532e724391830c0785d32eb2fb ({username='slave'} at 127.13.251.135:53413)
I20240501 20:09:41.223181 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet a115fe66d11143ea8e24aa891e4e2f33 from peer 744c5a532e724391830c0785d32eb2fb at {username='slave'} at 127.13.251.135:53413: session id = 744c5a532e724391830c0785d32eb2fb-a115fe66d11143ea8e24aa891e4e2f33
I20240501 20:09:41.226950 21118 tablet_copy_source_session.cc:205] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:41.229065 24858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a115fe66d11143ea8e24aa891e4e2f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.227239 20825 catalog_manager.cc:5484] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:41.237206 24858 tablet_copy_client.cc:783] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:41.237573 24858 tablet_copy_client.cc:647] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:41.240355 24858 tablet_copy_client.cc:523] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:41.244794 24858 tablet_bootstrap.cc:492] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:41.255582 24817 tablet_bootstrap.cc:492] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:41.256217 24817 tablet_bootstrap.cc:492] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: Bootstrap complete.
I20240501 20:09:41.256611 24817 ts_tablet_manager.cc:1367] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.131s user 0.057s sys 0.013s
I20240501 20:09:41.257753 24817 raft_consensus.cc:348] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:41.258152 24817 raft_consensus.cc:729] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.258584 24817 consensus_queue.cc:260] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:41.259385 24817 ts_tablet_manager.cc:1398] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:41.260701 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 744c5a532e724391830c0785d32eb2fb-30d2aada073b4d8989d7db8be4034756 received from {username='slave'} at 127.13.251.135:46111
I20240501 20:09:41.260843 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 744c5a532e724391830c0785d32eb2fb-30d2aada073b4d8989d7db8be4034756 on tablet 30d2aada073b4d8989d7db8be4034756 with peer 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:41.260960 24839 tablet_bootstrap.cc:492] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:41.261438 24839 tablet_bootstrap.cc:492] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:09:41.261752 24839 ts_tablet_manager.cc:1367] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.151s user 0.085s sys 0.019s
I20240501 20:09:41.262634 24839 raft_consensus.cc:348] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:41.262992 24839 raft_consensus.cc:729] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.263303 24839 consensus_queue.cc:260] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:41.263923 24839 ts_tablet_manager.cc:1398] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:41.264851 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-c1e295a46a174f3f8673e03f57890b58 received from {username='slave'} at 127.13.251.136:42501
I20240501 20:09:41.265028 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-c1e295a46a174f3f8673e03f57890b58 on tablet c1e295a46a174f3f8673e03f57890b58 with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:41.270354 24839 ts_tablet_manager.cc:902] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:41.271466 24839 tablet_copy_client.cc:308] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:09:41.272126 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet 43e80df9af0e40c5acc4dea95d7e95a8 from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:42501)
I20240501 20:09:41.272294 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet 43e80df9af0e40c5acc4dea95d7e95a8 from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:42501: session id = 8559229e044b4e1ea661b881b3f9af50-43e80df9af0e40c5acc4dea95d7e95a8
I20240501 20:09:41.274708 21248 tablet_copy_source_session.cc:205] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:41.275853 24839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e80df9af0e40c5acc4dea95d7e95a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.277176 20825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:41.283382 24839 tablet_copy_client.cc:783] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:41.283699 24839 tablet_copy_client.cc:647] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:41.285375 24839 tablet_copy_client.cc:523] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:41.286832 24832 tablet.cc:1722] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:41.288285 24843 tablet.cc:1722] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:41.290362 24839 tablet_bootstrap.cc:492] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:41.300194 20824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=24e4f27ebbb445bf8990672e3a971fef] alter complete (version 3)
I20240501 20:09:41.313138 24837 tablet.cc:1722] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:41.331796 20965 raft_consensus.cc:1203] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:09:41.351650 24858 tablet_bootstrap.cc:492] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:41.352298 24858 tablet_bootstrap.cc:492] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: Bootstrap complete.
I20240501 20:09:41.352730 24858 ts_tablet_manager.cc:1367] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.108s user 0.056s sys 0.005s
I20240501 20:09:41.354504 24858 raft_consensus.cc:348] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:41.354984 24858 raft_consensus.cc:729] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.355587 24858 consensus_queue.cc:260] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:41.356189 24858 ts_tablet_manager.cc:1398] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:41.357317 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session 744c5a532e724391830c0785d32eb2fb-a115fe66d11143ea8e24aa891e4e2f33 received from {username='slave'} at 127.13.251.135:53413
I20240501 20:09:41.357506 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session 744c5a532e724391830c0785d32eb2fb-a115fe66d11143ea8e24aa891e4e2f33 on tablet a115fe66d11143ea8e24aa891e4e2f33 with peer 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:41.362663 24839 tablet_bootstrap.cc:492] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:41.363060 24839 tablet_bootstrap.cc:492] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:09:41.363302 24839 ts_tablet_manager.cc:1367] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.073s user 0.049s sys 0.005s
I20240501 20:09:41.363991 24839 raft_consensus.cc:348] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:41.364297 24839 raft_consensus.cc:729] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.364596 24839 consensus_queue.cc:260] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:41.366241 24839 ts_tablet_manager.cc:1398] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240501 20:09:41.367658 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-43e80df9af0e40c5acc4dea95d7e95a8 received from {username='slave'} at 127.13.251.136:42501
I20240501 20:09:41.367802 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-43e80df9af0e40c5acc4dea95d7e95a8 on tablet 43e80df9af0e40c5acc4dea95d7e95a8 with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:41.407652 20822 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:41.407905 20822 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:41.411525 20822 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 16ab6a45179644e79032c298d309f79a
I20240501 20:09:41.412356 23956 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ab6a45179644e79032c298d309f79a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:41 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:41.412907 24893 tablet_replica.cc:330] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:41.412986 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ab6a45179644e79032c298d309f79a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:41 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:41.413214 24893 raft_consensus.cc:2229] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:41.413676 24893 raft_consensus.cc:2258] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:41.414703 24893 ts_tablet_manager.cc:1875] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:41.414861 23689 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ab6a45179644e79032c298d309f79a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:41 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:41.415392 24894 tablet_replica.cc:330] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:41.415650 24894 raft_consensus.cc:2229] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:41.415915 24894 raft_consensus.cc:2258] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:41.416688 24894 ts_tablet_manager.cc:1875] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:41.423259 24894 ts_tablet_manager.cc:1888] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:41.423378 24894 log.cc:1196] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/16ab6a45179644e79032c298d309f79a
I20240501 20:09:41.423885 24894 ts_tablet_manager.cc:1909] T 16ab6a45179644e79032c298d309f79a P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:41.425266 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 16ab6a45179644e79032c298d309f79a (table rebalancer_dynamic_table_new_name [id=24e4f27ebbb445bf8990672e3a971fef]) successfully deleted
I20240501 20:09:41.427937 24895 tablet_replica.cc:330] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:41.428201 24895 raft_consensus.cc:2229] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:41.428494 24895 raft_consensus.cc:2258] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:41.429428 24893 ts_tablet_manager.cc:1888] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:41.429565 24893 log.cc:1196] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/16ab6a45179644e79032c298d309f79a
I20240501 20:09:41.429553 24895 ts_tablet_manager.cc:1875] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:41.430084 24893 ts_tablet_manager.cc:1909] T 16ab6a45179644e79032c298d309f79a P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:41.431120 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 16ab6a45179644e79032c298d309f79a (table rebalancer_dynamic_table_new_name [id=24e4f27ebbb445bf8990672e3a971fef]) successfully deleted
I20240501 20:09:41.435487 24895 ts_tablet_manager.cc:1888] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:41.435629 24895 log.cc:1196] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/16ab6a45179644e79032c298d309f79a
I20240501 20:09:41.436154 24895 ts_tablet_manager.cc:1909] T 16ab6a45179644e79032c298d309f79a P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:41.436731 23977 raft_consensus.cc:1203] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:41.437254 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 16ab6a45179644e79032c298d309f79a (table rebalancer_dynamic_table_new_name [id=24e4f27ebbb445bf8990672e3a971fef]) successfully deleted
I20240501 20:09:41.454751 23689 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:41.460737 21336 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:09:41.470002 21207 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:41.495528 23956 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:41.516395 24089 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:41.520071 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:41.524065 24108 raft_consensus.cc:1203] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:41.530539 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef3b933ac3e4cae86faf4c21b18032f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2d555ca3d17248a0ba040f8e212a42f2]), partition=
I20240501 20:09:41.530961 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef3b933ac3e4cae86faf4c21b18032f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.535995 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef3b933ac3e4cae86faf4c21b18032f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2d555ca3d17248a0ba040f8e212a42f2]), partition=
I20240501 20:09:41.536168 24848 consensus_queue.cc:1035] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.536376 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef3b933ac3e4cae86faf4c21b18032f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.540369 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef3b933ac3e4cae86faf4c21b18032f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2d555ca3d17248a0ba040f8e212a42f2]), partition=
I20240501 20:09:41.540709 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef3b933ac3e4cae86faf4c21b18032f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.544052 23976 raft_consensus.cc:1203] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:41.558502 24911 tablet_bootstrap.cc:492] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:41.560510 24911 tablet_bootstrap.cc:654] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.569921 24915 tablet_bootstrap.cc:492] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:41.572243 24915 tablet_bootstrap.cc:654] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.574921 24911 tablet_bootstrap.cc:492] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:41.575204 24911 ts_tablet_manager.cc:1367] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240501 20:09:41.576339 24911 raft_consensus.cc:348] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.576725 24911 raft_consensus.cc:374] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.576910 24911 raft_consensus.cc:729] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.577266 24911 consensus_queue.cc:260] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.579082 24911 ts_tablet_manager.cc:1398] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:09:41.585132 24848 consensus_queue.cc:1035] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.587486 24912 tablet_bootstrap.cc:492] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:41.593492 24912 tablet_bootstrap.cc:654] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.596642 24915 tablet_bootstrap.cc:492] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:41.596809 24915 ts_tablet_manager.cc:1367] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.004s
I20240501 20:09:41.597577 24915 raft_consensus.cc:348] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.597795 24915 raft_consensus.cc:374] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.597898 24915 raft_consensus.cc:729] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.601794 24915 consensus_queue.cc:260] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.602578 24915 ts_tablet_manager.cc:1398] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240501 20:09:41.607260 24783 consensus_queue.cc:1035] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.609670 20945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:41.609023 24912 tablet_bootstrap.cc:492] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:41.610347 24912 ts_tablet_manager.cc:1367] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240501 20:09:41.611595 24912 raft_consensus.cc:348] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.612470 24912 raft_consensus.cc:374] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.612592 24912 raft_consensus.cc:729] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.612833 24912 consensus_queue.cc:260] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.616323 24912 ts_tablet_manager.cc:1398] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20240501 20:09:41.628228 21162 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 68449.3us
I20240501 20:09:41.644866 24866 consensus_queue.cc:1035] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.658273 23825 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:41.659482 24217 consensus_queue.cc:1035] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.666154 24217 consensus_queue.cc:1035] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.675974 21076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:41.745621 20825 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:41.754889 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 692fd849cd99422b8cb7fc4d202d8ab5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.755403 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 692fd849cd99422b8cb7fc4d202d8ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.755378 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 692fd849cd99422b8cb7fc4d202d8ab5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.756021 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 692fd849cd99422b8cb7fc4d202d8ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.756040 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 692fd849cd99422b8cb7fc4d202d8ab5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.756474 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 692fd849cd99422b8cb7fc4d202d8ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.758635 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 9d308589b223471bb2f20ff3c6a7f13e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.759074 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d308589b223471bb2f20ff3c6a7f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.759135 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 9d308589b223471bb2f20ff3c6a7f13e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.759528 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d308589b223471bb2f20ff3c6a7f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.762750 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 9d308589b223471bb2f20ff3c6a7f13e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.763131 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d308589b223471bb2f20ff3c6a7f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.763080 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3c1d24969140d2a5996ee27aead916 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.763460 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3c1d24969140d2a5996ee27aead916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.763499 24915 tablet_bootstrap.cc:492] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:41.765074 24915 tablet_bootstrap.cc:654] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.767382 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3c1d24969140d2a5996ee27aead916 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.767832 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3c1d24969140d2a5996ee27aead916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.770025 24911 tablet_bootstrap.cc:492] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:41.770134 24108 raft_consensus.cc:1203] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:41.770275 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3c1d24969140d2a5996ee27aead916 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.770650 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3c1d24969140d2a5996ee27aead916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.771631 24911 tablet_bootstrap.cc:654] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.778268 24931 tablet_bootstrap.cc:492] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:41.779788 24915 tablet_bootstrap.cc:492] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:41.780021 24915 ts_tablet_manager.cc:1367] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240501 20:09:41.780202 24931 tablet_bootstrap.cc:654] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.781064 24915 raft_consensus.cc:348] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.781399 24915 raft_consensus.cc:374] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.781498 24915 raft_consensus.cc:729] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.782055 24915 consensus_queue.cc:260] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.785238 24915 ts_tablet_manager.cc:1398] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:41.785924 24911 tablet_bootstrap.cc:492] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:41.786162 24911 ts_tablet_manager.cc:1367] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.011s
I20240501 20:09:41.786302 24930 tablet_bootstrap.cc:492] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:41.787015 24911 raft_consensus.cc:348] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.787266 24911 raft_consensus.cc:374] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.787346 24911 raft_consensus.cc:729] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.787659 24911 consensus_queue.cc:260] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.788550 24930 tablet_bootstrap.cc:654] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.790875 24911 ts_tablet_manager.cc:1398] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:41.794653 24911 tablet_bootstrap.cc:492] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:41.797916 24911 tablet_bootstrap.cc:654] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.800683 24931 tablet_bootstrap.cc:492] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:41.801206 24931 ts_tablet_manager.cc:1367] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.004s
I20240501 20:09:41.803637 24936 tablet_bootstrap.cc:492] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:41.802366 24931 raft_consensus.cc:348] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:41.804042 24931 raft_consensus.cc:374] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.804416 24931 raft_consensus.cc:729] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.805063 24930 tablet_bootstrap.cc:492] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:41.805305 24930 ts_tablet_manager.cc:1367] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20240501 20:09:41.806210 24936 tablet_bootstrap.cc:654] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.806419 24930 raft_consensus.cc:348] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.806566 24931 consensus_queue.cc:260] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:41.806809 24930 raft_consensus.cc:374] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.806999 24930 raft_consensus.cc:729] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.807224 24930 consensus_queue.cc:260] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.808640 24911 tablet_bootstrap.cc:492] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:41.808629 24930 ts_tablet_manager.cc:1398] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:41.808835 24911 ts_tablet_manager.cc:1367] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20240501 20:09:41.808907 24930 tablet_bootstrap.cc:492] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:41.809875 24931 ts_tablet_manager.cc:1398] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 20:09:41.810773 24930 tablet_bootstrap.cc:654] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.811213 24911 raft_consensus.cc:348] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.811506 24911 raft_consensus.cc:374] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.811673 24911 raft_consensus.cc:729] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.811862 24931 tablet_bootstrap.cc:492] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:41.813798 24911 consensus_queue.cc:260] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.817198 24931 tablet_bootstrap.cc:654] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.819770 24911 ts_tablet_manager.cc:1398] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:09:41.819865 24930 tablet_bootstrap.cc:492] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:41.820079 24930 ts_tablet_manager.cc:1367] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240501 20:09:41.820699 24936 tablet_bootstrap.cc:492] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:41.821007 24934 tablet_bootstrap.cc:492] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:41.821893 24936 ts_tablet_manager.cc:1367] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240501 20:09:41.822824 24934 tablet_bootstrap.cc:654] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.822762 24936 raft_consensus.cc:348] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:41.823021 24936 raft_consensus.cc:374] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.823168 24936 raft_consensus.cc:729] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.823491 24936 consensus_queue.cc:260] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:41.823810 24930 raft_consensus.cc:348] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:41.824050 24930 raft_consensus.cc:374] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.824131 24930 raft_consensus.cc:729] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.824364 24930 consensus_queue.cc:260] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:41.824894 24930 ts_tablet_manager.cc:1398] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:09:41.826274 24936 ts_tablet_manager.cc:1398] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:41.828235 24931 tablet_bootstrap.cc:492] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:41.828397 24931 ts_tablet_manager.cc:1367] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20240501 20:09:41.829151 24931 raft_consensus.cc:348] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.831647 24931 raft_consensus.cc:374] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.831729 24931 raft_consensus.cc:729] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.831959 24931 consensus_queue.cc:260] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.832499 24931 ts_tablet_manager.cc:1398] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:41.833658 24934 tablet_bootstrap.cc:492] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:41.833835 24934 ts_tablet_manager.cc:1367] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:09:41.834790 24934 raft_consensus.cc:348] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.835000 24934 raft_consensus.cc:374] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.835314 24934 raft_consensus.cc:729] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.835669 24934 consensus_queue.cc:260] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.836357 24934 ts_tablet_manager.cc:1398] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:09:41.866122 24472 raft_consensus.cc:482] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.866317 24472 raft_consensus.cc:504] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.867151 24472 leader_election.cc:288] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:41.867579 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef3b933ac3e4cae86faf4c21b18032f" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:41.867631 24923 raft_consensus.cc:482] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.867831 20966 raft_consensus.cc:2454] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:41.867785 24923 raft_consensus.cc:504] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.867959 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef3b933ac3e4cae86faf4c21b18032f" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:41.868189 23845 raft_consensus.cc:2454] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:41.868304 23909 leader_election.cc:302] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:41.868531 24472 raft_consensus.cc:2781] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.868636 24472 raft_consensus.cc:482] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.868737 24472 raft_consensus.cc:3037] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.868788 24923 leader_election.cc:288] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:41.871619 24472 raft_consensus.cc:504] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.872709 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef3b933ac3e4cae86faf4c21b18032f" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:41.872737 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef3b933ac3e4cae86faf4c21b18032f" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:41.872900 23845 raft_consensus.cc:3037] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.872939 20966 raft_consensus.cc:3037] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.875720 20966 raft_consensus.cc:2454] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:41.876024 23845 raft_consensus.cc:2454] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:41.876158 23909 leader_election.cc:302] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:41.876811 24472 leader_election.cc:288] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:41.876873 24948 raft_consensus.cc:2781] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.879413 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3c1d24969140d2a5996ee27aead916" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:41.879678 23712 raft_consensus.cc:2454] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:41.879768 24866 raft_consensus.cc:482] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.879947 24866 raft_consensus.cc:504] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:41.880200 21163 leader_election.cc:302] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46; no voters:
I20240501 20:09:41.880514 24923 raft_consensus.cc:2781] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.880618 24923 raft_consensus.cc:482] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.880692 24923 raft_consensus.cc:3037] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.880832 24866 leader_election.cc:288] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:41.884346 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d308589b223471bb2f20ff3c6a7f13e" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:41.884608 24108 raft_consensus.cc:2454] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:41.885813 21033 leader_election.cc:302] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:41.886099 24866 raft_consensus.cc:2781] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.886183 24866 raft_consensus.cc:482] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.886293 24866 raft_consensus.cc:3037] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.887845 24948 raft_consensus.cc:686] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:41.888295 24948 consensus_queue.cc:237] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.888553 24866 raft_consensus.cc:504] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:41.889142 24866 leader_election.cc:288] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:41.890544 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d308589b223471bb2f20ff3c6a7f13e" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:41.890832 24108 raft_consensus.cc:3037] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.894578 24108 raft_consensus.cc:2454] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:41.895453 21033 leader_election.cc:302] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:41.896356 24787 raft_consensus.cc:2781] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.896497 24787 raft_consensus.cc:686] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:41.896854 24787 consensus_queue.cc:237] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:41.898561 20825 catalog_manager.cc:5484] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:41.902752 24313 raft_consensus.cc:482] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.902910 24313 raft_consensus.cc:504] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.903589 24313 leader_election.cc:288] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:41.905963 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3c1d24969140d2a5996ee27aead916" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:41.906242 23712 raft_consensus.cc:2454] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:41.906807 23781 leader_election.cc:302] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:41.907168 24313 raft_consensus.cc:2781] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.907286 24313 raft_consensus.cc:482] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.907392 24313 raft_consensus.cc:3037] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.907845 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3c1d24969140d2a5996ee27aead916" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:41.908303 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d308589b223471bb2f20ff3c6a7f13e" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:41.908568 21226 raft_consensus.cc:2454] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:41.909897 24313 raft_consensus.cc:504] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.910583 24313 leader_election.cc:288] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:41.910842 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d308589b223471bb2f20ff3c6a7f13e" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:41.911103 21227 raft_consensus.cc:3037] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.911028 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3c1d24969140d2a5996ee27aead916" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:41.912382 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3c1d24969140d2a5996ee27aead916" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:41.912592 23712 raft_consensus.cc:3037] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.913388 24923 raft_consensus.cc:504] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.913803 21227 raft_consensus.cc:2454] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:41.914222 24923 leader_election.cc:288] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:41.915129 23712 raft_consensus.cc:2454] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:41.915167 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3c1d24969140d2a5996ee27aead916" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:41.915364 23711 raft_consensus.cc:2379] T 6e3c1d24969140d2a5996ee27aead916 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4af096f5a6a9440a94f6682b399efe20 in current term 1: Already voted for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in this term.
I20240501 20:09:41.915457 21228 raft_consensus.cc:2379] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate 4af096f5a6a9440a94f6682b399efe20 in this term.
I20240501 20:09:41.916373 23781 leader_election.cc:302] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:09:41.916702 24313 raft_consensus.cc:2781] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.917361 24313 raft_consensus.cc:686] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:41.917740 24313 consensus_queue.cc:237] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.921154 20825 catalog_manager.cc:5484] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:41.925455 20825 catalog_manager.cc:5484] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:41.931495 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3c1d24969140d2a5996ee27aead916" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:41.933547 24848 raft_consensus.cc:482] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.933728 24848 raft_consensus.cc:504] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.934391 24848 leader_election.cc:288] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:41.934738 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3c1d24969140d2a5996ee27aead916" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:41.935214 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "692fd849cd99422b8cb7fc4d202d8ab5" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:41.935405 23977 raft_consensus.cc:2454] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:41.935859 24043 leader_election.cc:302] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:41.936245 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "692fd849cd99422b8cb7fc4d202d8ab5" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:41.936456 23844 raft_consensus.cc:2454] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:41.936503 21164 leader_election.cc:302] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:41.937116 24848 raft_consensus.cc:2781] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.937117 24923 raft_consensus.cc:2726] T 6e3c1d24969140d2a5996ee27aead916 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:09:41.937240 24848 raft_consensus.cc:482] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.937352 24848 raft_consensus.cc:3037] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.940021 24848 raft_consensus.cc:504] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.941090 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "692fd849cd99422b8cb7fc4d202d8ab5" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:41.941329 23977 raft_consensus.cc:3037] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.943677 23977 raft_consensus.cc:2454] T 692fd849cd99422b8cb7fc4d202d8ab5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:41.943782 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "692fd849cd99422b8cb7fc4d202d8ab5" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:41.943990 23844 raft_consensus.cc:3037] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.944312 24043 leader_election.cc:302] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:41.944599 24217 raft_consensus.cc:2781] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.949142 23844 raft_consensus.cc:2454] T 692fd849cd99422b8cb7fc4d202d8ab5 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:41.952821 24848 leader_election.cc:288] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:41.956880 24217 raft_consensus.cc:686] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:41.957221 24217 consensus_queue.cc:237] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:41.961230 20825 catalog_manager.cc:5484] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:42.028848 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:42.031325 23712 consensus_queue.cc:237] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:42.035729 24108 raft_consensus.cc:1261] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.036442 24667 consensus_queue.cc:1035] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.039007 23844 raft_consensus.cc:1261] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.041061 24949 consensus_queue.cc:1035] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.042809 20966 raft_consensus.cc:1261] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.043648 24949 consensus_queue.cc:1035] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:42.044854 23645 consensus_peers.cc:485] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 939b60669869461483f50e61d8dc633b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.047837 23845 raft_consensus.cc:1261] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.048882 24667 consensus_queue.cc:1035] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.055601 20823 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 1ef3b933ac3e4cae86faf4c21b18032f (table rebalancer_dynamic_table [id=2d555ca3d17248a0ba040f8e212a42f2]). Expected version 1 got 0
I20240501 20:09:42.065846 24974 tablet.cc:1722] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:42.067970 24974 tablet.cc:1620] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:42.069768 20823 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 1ef3b933ac3e4cae86faf4c21b18032f (table rebalancer_dynamic_table [id=2d555ca3d17248a0ba040f8e212a42f2]). Expected version 1 got 0
I20240501 20:09:42.075477 24667 raft_consensus.cc:2932] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:42.076139 24108 raft_consensus.cc:2932] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:42.081290 24973 tablet.cc:1722] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:42.081936 24973 tablet.cc:1620] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:42.081591 23845 raft_consensus.cc:2932] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:42.082170 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=2d555ca3d17248a0ba040f8e212a42f2] alter complete (version 1)
I20240501 20:09:42.086457 20823 catalog_manager.cc:5484] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:42.094776 20825 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 1ef3b933ac3e4cae86faf4c21b18032f (table rebalancer_dynamic_table [id=2d555ca3d17248a0ba040f8e212a42f2]). Expected version 1 got 0
I20240501 20:09:42.099696 24981 tablet.cc:1722] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:42.100597 24981 tablet.cc:1620] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:42.101157 20825 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 1ef3b933ac3e4cae86faf4c21b18032f (table rebalancer_dynamic_table [id=2d555ca3d17248a0ba040f8e212a42f2]). Expected version 1 got 0
I20240501 20:09:42.135485 21358 consensus_queue.cc:237] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.137840 21228 raft_consensus.cc:1261] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:42.137851 20966 raft_consensus.cc:1261] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:09:42.138406 21291 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Couldn't send request to peer 744c5a532e724391830c0785d32eb2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4587a9d5392742b58ec8d64619f2dda1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.138446 24995 consensus_queue.cc:1035] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:42.138798 24171 consensus_queue.cc:1035] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:42.141676 24995 raft_consensus.cc:2932] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:42.142383 21228 raft_consensus.cc:2932] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:42.148214 20825 catalog_manager.cc:5484] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:42.150821 20966 raft_consensus.cc:2932] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:42.188105 20966 consensus_queue.cc:237] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.190603 21358 raft_consensus.cc:1261] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:42.191087 21228 raft_consensus.cc:1261] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:42.191046 20824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:42.191237 24634 consensus_queue.cc:1035] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:42.191785 25015 consensus_queue.cc:1035] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:09:42.192487 20902 consensus_peers.cc:485] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a2f74307a194dc49d184d4ea288b36d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.194532 24634 raft_consensus.cc:2932] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:42.196139 21228 raft_consensus.cc:2932] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:42.200963 21358 raft_consensus.cc:2932] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:42.202292 24973 tablet.cc:1722] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:42.202761 24973 tablet.cc:1620] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:42.205406 24974 tablet.cc:1722] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:42.205849 24974 tablet.cc:1620] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:42.209146 20825 catalog_manager.cc:5484] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:42.210616 24981 tablet.cc:1722] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:42.211194 24981 tablet.cc:1620] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:42.216094 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=2d555ca3d17248a0ba040f8e212a42f2] alter complete (version 2)
I20240501 20:09:42.252322 20966 consensus_queue.cc:237] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:42.254673 23977 raft_consensus.cc:1261] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.254709 24108 raft_consensus.cc:1261] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:09:42.254887 20900 consensus_peers.cc:485] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68133e033c824e11a1ed744bff76a934. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.255227 24634 consensus_queue.cc:1035] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.255519 25015 consensus_queue.cc:1035] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.258191 24634 raft_consensus.cc:2932] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:42.258754 23977 raft_consensus.cc:2932] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:42.261690 24108 raft_consensus.cc:2932] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:42.263314 20824 catalog_manager.cc:5484] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:42.279618 24787 consensus_queue.cc:1035] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.288880 24783 consensus_queue.cc:1035] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.313313 23845 consensus_queue.cc:237] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:42.316129 24108 raft_consensus.cc:1261] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.316402 21228 raft_consensus.cc:1261] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.317070 24958 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.317778 24958 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.318897 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:42.329221 24973 tablet.cc:1722] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:42.332345 24981 tablet.cc:1722] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:42.342602 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=2d555ca3d17248a0ba040f8e212a42f2] alter complete (version 3)
I20240501 20:09:42.348759 24974 tablet.cc:1722] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:42.355898 24958 raft_consensus.cc:2932] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
W20240501 20:09:42.357702 23780 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed53558d93214d34aedca8505870ec61. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.361302 24108 raft_consensus.cc:2932] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:42.361318 21227 raft_consensus.cc:2932] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:42.367599 20820 catalog_manager.cc:5484] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:42.367868 25058 consensus_queue.cc:1035] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.373489 25058 consensus_queue.cc:1035] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.373950 24217 consensus_queue.cc:1035] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.399726 24217 consensus_queue.cc:1035] T 692fd849cd99422b8cb7fc4d202d8ab5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.419678 23712 consensus_queue.cc:237] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:42.422483 20966 raft_consensus.cc:1261] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:09:42.422523 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4715a768081a49228ab47b8071c40c8f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.423331 23845 raft_consensus.cc:1261] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.423298 24968 consensus_queue.cc:1035] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.423890 24667 consensus_queue.cc:1035] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.426445 24968 raft_consensus.cc:2932] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:42.427037 23845 raft_consensus.cc:2932] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:42.431854 20966 raft_consensus.cc:2932] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:42.432816 20823 catalog_manager.cc:5484] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:42.450138 20824 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:42.450441 20824 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:42.454612 25090 ts_tablet_manager.cc:902] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:42.455153 20824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 1ef3b933ac3e4cae86faf4c21b18032f
I20240501 20:09:42.456511 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef3b933ac3e4cae86faf4c21b18032f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:42 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:42.457269 25100 tablet_replica.cc:330] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:42.457489 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef3b933ac3e4cae86faf4c21b18032f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:42 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:42.457541 25100 raft_consensus.cc:2229] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:42.457854 25100 raft_consensus.cc:2258] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:42.457942 23956 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef3b933ac3e4cae86faf4c21b18032f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:42 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:42.459272 25100 ts_tablet_manager.cc:1875] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:42.460727 25102 tablet_replica.cc:330] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:42.462265 25102 raft_consensus.cc:2229] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:42.463161 25102 raft_consensus.cc:2258] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:42.465482 25102 ts_tablet_manager.cc:1875] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:42.472307 25100 ts_tablet_manager.cc:1888] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:42.472543 25100 log.cc:1196] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/1ef3b933ac3e4cae86faf4c21b18032f
I20240501 20:09:42.473309 25100 ts_tablet_manager.cc:1909] T 1ef3b933ac3e4cae86faf4c21b18032f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:09:42.474500 25090 tablet_copy_client.cc:308] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.133:37155
I20240501 20:09:42.475981 23734 tablet_copy_service.cc:140] P 51bf640e895b4e5eb91bdf7509ea7e46: Received BeginTabletCopySession request for tablet 939b60669869461483f50e61d8dc633b from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:45435)
I20240501 20:09:42.475951 20824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:42.476202 23734 tablet_copy_service.cc:161] P 51bf640e895b4e5eb91bdf7509ea7e46: Beginning new tablet copy session on tablet 939b60669869461483f50e61d8dc633b from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:45435: session id = 61380023001a4ee9ada9ec2c6c1b7159-939b60669869461483f50e61d8dc633b
I20240501 20:09:42.478025 25102 ts_tablet_manager.cc:1888] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:42.478150 25102 log.cc:1196] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/1ef3b933ac3e4cae86faf4c21b18032f
I20240501 20:09:42.479290 25102 ts_tablet_manager.cc:1909] T 1ef3b933ac3e4cae86faf4c21b18032f P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:42.479691 23734 tablet_copy_source_session.cc:205] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:42.480389 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 1ef3b933ac3e4cae86faf4c21b18032f (table rebalancer_dynamic_table_new_name [id=2d555ca3d17248a0ba040f8e212a42f2]) successfully deleted
I20240501 20:09:42.480530 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 1ef3b933ac3e4cae86faf4c21b18032f (table rebalancer_dynamic_table_new_name [id=2d555ca3d17248a0ba040f8e212a42f2]) successfully deleted
I20240501 20:09:42.483176 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939b60669869461483f50e61d8dc633b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.486683 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 0fda032f2be744fe95170b3c3287d6ae (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.487111 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fda032f2be744fe95170b3c3287d6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.487143 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd04e795cbd4646beefd20cd7a38418 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.487203 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd04e795cbd4646beefd20cd7a38418 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.487567 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd04e795cbd4646beefd20cd7a38418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.487684 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd04e795cbd4646beefd20cd7a38418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.491797 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 0fda032f2be744fe95170b3c3287d6ae (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.492206 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fda032f2be744fe95170b3c3287d6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.494110 23955 tablet_service.cc:1467] Processing CreateTablet for tablet d37fde63d6414745b0f81c0a0abe4f46 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.494683 23955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37fde63d6414745b0f81c0a0abe4f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.497242 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd04e795cbd4646beefd20cd7a38418 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.497753 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd04e795cbd4646beefd20cd7a38418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.498157 25090 tablet_copy_client.cc:783] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:42.498445 25090 tablet_copy_client.cc:647] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:42.499977 24089 tablet_service.cc:1467] Processing CreateTablet for tablet d37fde63d6414745b0f81c0a0abe4f46 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.503345 23825 tablet_service.cc:1467] Processing CreateTablet for tablet d37fde63d6414745b0f81c0a0abe4f46 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.503409 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 0fda032f2be744fe95170b3c3287d6ae (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.503841 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37fde63d6414745b0f81c0a0abe4f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.503844 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fda032f2be744fe95170b3c3287d6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.507604 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37fde63d6414745b0f81c0a0abe4f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.512780 25109 tablet_bootstrap.cc:492] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:42.514494 25109 tablet_bootstrap.cc:654] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.525928 25109 tablet_bootstrap.cc:492] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:42.526590 25109 ts_tablet_manager.cc:1367] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240501 20:09:42.528776 25090 tablet_copy_client.cc:523] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:42.529490 25109 raft_consensus.cc:348] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:42.529989 25109 raft_consensus.cc:374] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.530221 25109 raft_consensus.cc:729] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.530726 25109 consensus_queue.cc:260] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:42.531996 25109 ts_tablet_manager.cc:1398] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:42.534471 25090 tablet_bootstrap.cc:492] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:42.543627 25111 tablet_bootstrap.cc:492] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:42.549396 25113 tablet_bootstrap.cc:492] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:42.551977 25112 tablet_bootstrap.cc:492] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:42.552502 25111 tablet_bootstrap.cc:654] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.554858 25107 tablet_bootstrap.cc:492] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:42.555855 25112 tablet_bootstrap.cc:654] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.557257 25113 tablet_bootstrap.cc:654] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.557847 25101 tablet_replica.cc:330] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:42.558076 25101 raft_consensus.cc:2229] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:42.558382 25101 raft_consensus.cc:2258] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:42.559701 25101 ts_tablet_manager.cc:1875] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:42.563566 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:42.569481 25101 ts_tablet_manager.cc:1888] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:42.569701 25101 log.cc:1196] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/1ef3b933ac3e4cae86faf4c21b18032f
I20240501 20:09:42.570261 25101 ts_tablet_manager.cc:1909] T 1ef3b933ac3e4cae86faf4c21b18032f P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:42.572633 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 1ef3b933ac3e4cae86faf4c21b18032f (table rebalancer_dynamic_table_new_name [id=2d555ca3d17248a0ba040f8e212a42f2]) successfully deleted
I20240501 20:09:42.575925 25090 tablet_bootstrap.cc:492] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:42.576581 25090 tablet_bootstrap.cc:492] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:09:42.576896 25090 ts_tablet_manager.cc:1367] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.000s
I20240501 20:09:42.577802 25090 raft_consensus.cc:348] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:42.578336 25090 raft_consensus.cc:729] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.578650 25090 consensus_queue.cc:260] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:42.578861 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef0087e992043daa170b474d11a142b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70d6282b175e476aa7efeb98d8e6c7fb]), partition=
I20240501 20:09:42.579219 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef0087e992043daa170b474d11a142b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.579342 25090 ts_tablet_manager.cc:1398] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:42.581284 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef0087e992043daa170b474d11a142b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70d6282b175e476aa7efeb98d8e6c7fb]), partition=
I20240501 20:09:42.581676 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef0087e992043daa170b474d11a142b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.582357 23734 tablet_copy_service.cc:342] P 51bf640e895b4e5eb91bdf7509ea7e46: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-939b60669869461483f50e61d8dc633b received from {username='slave'} at 127.13.251.132:45435
I20240501 20:09:42.582561 23734 tablet_copy_service.cc:433] P 51bf640e895b4e5eb91bdf7509ea7e46: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-939b60669869461483f50e61d8dc633b on tablet 939b60669869461483f50e61d8dc633b with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:09:42.588651 25109 tablet_bootstrap.cc:492] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:42.578238 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef0087e992043daa170b474d11a142b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70d6282b175e476aa7efeb98d8e6c7fb]), partition=
I20240501 20:09:42.589771 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef0087e992043daa170b474d11a142b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.590229 25109 tablet_bootstrap.cc:654] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.600039 25107 tablet_bootstrap.cc:654] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.607951 25108 tablet_bootstrap.cc:492] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:42.608490 25109 tablet_bootstrap.cc:492] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:42.608747 25109 ts_tablet_manager.cc:1367] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.009s
I20240501 20:09:42.609566 25109 raft_consensus.cc:348] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:42.609859 25109 raft_consensus.cc:374] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.609977 25109 raft_consensus.cc:729] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.610276 25109 consensus_queue.cc:260] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:42.610939 25108 tablet_bootstrap.cc:654] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.611869 25109 ts_tablet_manager.cc:1398] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:42.629253 25112 tablet_bootstrap.cc:492] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:42.629518 25112 ts_tablet_manager.cc:1367] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.000s
I20240501 20:09:42.630342 25112 raft_consensus.cc:348] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.630630 25112 raft_consensus.cc:374] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.628860 25111 tablet_bootstrap.cc:492] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:42.630743 25112 raft_consensus.cc:729] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.630997 25112 consensus_queue.cc:260] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.632678 25112 ts_tablet_manager.cc:1398] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:42.633014 25112 tablet_bootstrap.cc:492] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:42.634598 25112 tablet_bootstrap.cc:654] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.637120 25112 tablet_bootstrap.cc:492] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:42.637331 25112 ts_tablet_manager.cc:1367] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240501 20:09:42.638631 25112 raft_consensus.cc:348] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:42.638911 25112 raft_consensus.cc:374] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.639101 25112 raft_consensus.cc:729] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.639458 25112 consensus_queue.cc:260] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:42.640180 25112 ts_tablet_manager.cc:1398] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 20:09:42.640489 25112 tablet_bootstrap.cc:492] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:42.642172 25112 tablet_bootstrap.cc:654] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.647625 25058 raft_consensus.cc:482] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:42.647781 25058 raft_consensus.cc:504] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:42.648680 25058 leader_election.cc:288] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:42.649680 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef0087e992043daa170b474d11a142b" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
W20240501 20:09:42.650553 23781 leader_election.cc:341] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:42.664561 25126 ts_tablet_manager.cc:902] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:42.669088 23976 consensus_queue.cc:237] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
W20240501 20:09:42.673691 23909 consensus_peers.cc:485] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30eb2ed6c8b64204b4c3a11b41e53ec8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.675181 24110 raft_consensus.cc:1261] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.676091 24949 consensus_queue.cc:1035] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.701664 25113 tablet_bootstrap.cc:492] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:42.701928 25107 tablet_bootstrap.cc:492] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:42.701977 25113 ts_tablet_manager.cc:1367] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.153s user 0.051s sys 0.048s
I20240501 20:09:42.674849 23712 raft_consensus.cc:1261] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.631987 25111 ts_tablet_manager.cc:1367] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.000s
I20240501 20:09:42.703070 25113 raft_consensus.cc:348] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.703390 25113 raft_consensus.cc:374] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.703570 25113 raft_consensus.cc:729] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.703164 24949 consensus_queue.cc:1035] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.703974 25113 consensus_queue.cc:260] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.704800 25113 ts_tablet_manager.cc:1398] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:42.707402 25134 ts_tablet_manager.cc:902] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:42.708040 24948 raft_consensus.cc:2932] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:42.716004 24110 raft_consensus.cc:2932] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:42.709005 23710 raft_consensus.cc:2932] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:42.723917 20824 catalog_manager.cc:5484] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:42.724365 25090 ts_tablet_manager.cc:902] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:42.727834 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef0087e992043daa170b474d11a142b" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
W20240501 20:09:42.728740 23779 leader_election.cc:341] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:42.728920 23779 leader_election.cc:302] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:42.729492 25058 raft_consensus.cc:2726] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:42.754369 25107 ts_tablet_manager.cc:1367] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.200s user 0.005s sys 0.000s
I20240501 20:09:42.755344 25112 tablet_bootstrap.cc:492] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:42.755498 25112 ts_tablet_manager.cc:1367] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.115s user 0.029s sys 0.065s
I20240501 20:09:42.756321 25112 raft_consensus.cc:348] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:42.756583 25112 raft_consensus.cc:374] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.756667 25112 raft_consensus.cc:729] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.756898 25112 consensus_queue.cc:260] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:42.757509 25112 ts_tablet_manager.cc:1398] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:42.759788 25111 raft_consensus.cc:348] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.761615 25108 tablet_bootstrap.cc:492] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:42.761792 25108 ts_tablet_manager.cc:1367] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.154s user 0.004s sys 0.000s
I20240501 20:09:42.763329 25111 raft_consensus.cc:374] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.757894 25107 raft_consensus.cc:348] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.766265 25107 raft_consensus.cc:374] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.766422 25107 raft_consensus.cc:729] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.766805 25107 consensus_queue.cc:260] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.767659 25111 raft_consensus.cc:729] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.771474 25111 consensus_queue.cc:260] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.771874 25126 tablet_copy_client.cc:308] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:41259
I20240501 20:09:42.773211 20987 tablet_copy_service.cc:140] P cda19d6a81c0473db302535ce2ada7f0: Received BeginTabletCopySession request for tablet 68133e033c824e11a1ed744bff76a934 from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:42109)
I20240501 20:09:42.773504 20987 tablet_copy_service.cc:161] P cda19d6a81c0473db302535ce2ada7f0: Beginning new tablet copy session on tablet 68133e033c824e11a1ed744bff76a934 from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:42109: session id = feac73db0a3b4aafa374ca5dfd322b08-68133e033c824e11a1ed744bff76a934
I20240501 20:09:42.780315 20987 tablet_copy_source_session.cc:205] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:42.783313 25126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68133e033c824e11a1ed744bff76a934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.791146 25129 ts_tablet_manager.cc:902] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:42.792028 25134 tablet_copy_client.cc:308] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:41259
I20240501 20:09:42.805667 20987 tablet_copy_service.cc:140] P cda19d6a81c0473db302535ce2ada7f0: Received BeginTabletCopySession request for tablet 7a2f74307a194dc49d184d4ea288b36d from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:53673)
I20240501 20:09:42.806126 20987 tablet_copy_service.cc:161] P cda19d6a81c0473db302535ce2ada7f0: Beginning new tablet copy session on tablet 7a2f74307a194dc49d184d4ea288b36d from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:53673: session id = 51bf640e895b4e5eb91bdf7509ea7e46-7a2f74307a194dc49d184d4ea288b36d
I20240501 20:09:42.810762 20987 tablet_copy_source_session.cc:205] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:42.812867 25134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2f74307a194dc49d184d4ea288b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.822492 25134 tablet_copy_client.cc:783] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:42.822935 25134 tablet_copy_client.cc:647] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:42.825712 25107 ts_tablet_manager.cc:1398] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.070s user 0.002s sys 0.000s
I20240501 20:09:42.826104 25107 tablet_bootstrap.cc:492] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:42.762730 25108 raft_consensus.cc:348] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.831967 25108 raft_consensus.cc:374] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.832079 25108 raft_consensus.cc:729] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.832386 25108 consensus_queue.cc:260] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.833039 25108 ts_tablet_manager.cc:1398] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.071s user 0.000s sys 0.002s
I20240501 20:09:42.833667 25107 tablet_bootstrap.cc:654] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.836737 25111 ts_tablet_manager.cc:1398] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.134s user 0.003s sys 0.000s
I20240501 20:09:42.845250 25107 tablet_bootstrap.cc:492] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:42.845482 25107 ts_tablet_manager.cc:1367] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 20:09:42.845945 25126 tablet_copy_client.cc:783] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:42.846362 25107 raft_consensus.cc:348] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.846689 25107 raft_consensus.cc:374] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.846802 25107 raft_consensus.cc:729] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.847173 25107 consensus_queue.cc:260] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.847913 25107 ts_tablet_manager.cc:1398] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:42.848278 25107 tablet_bootstrap.cc:492] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:42.848443 25126 tablet_copy_client.cc:647] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:42.850255 25107 tablet_bootstrap.cc:654] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.867022 25137 raft_consensus.cc:482] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:42.867218 25137 raft_consensus.cc:504] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.868191 25137 leader_election.cc:288] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:42.868995 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd04e795cbd4646beefd20cd7a38418" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:42.869237 21357 raft_consensus.cc:2454] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:42.869388 25134 tablet_copy_client.cc:523] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:42.869604 25126 tablet_copy_client.cc:523] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:42.869714 23910 leader_election.cc:302] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:42.870127 24968 raft_consensus.cc:482] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:42.870268 24968 raft_consensus.cc:504] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.870594 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd04e795cbd4646beefd20cd7a38418" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:42.870769 25137 raft_consensus.cc:2781] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:42.870800 24110 raft_consensus.cc:2454] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:42.870878 25137 raft_consensus.cc:482] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:42.870971 25137 raft_consensus.cc:3037] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.870990 24968 leader_election.cc:288] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:42.871945 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda032f2be744fe95170b3c3287d6ae" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:42.872177 23977 raft_consensus.cc:2454] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:42.872304 25058 raft_consensus.cc:482] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:42.872450 25058 raft_consensus.cc:504] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:42.873178 25058 leader_election.cc:288] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:42.873322 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda032f2be744fe95170b3c3287d6ae" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:42.873553 21228 raft_consensus.cc:2454] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:42.874006 23647 leader_election.cc:302] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46; no voters:
I20240501 20:09:42.874400 24968 raft_consensus.cc:2781] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:42.874511 24968 raft_consensus.cc:482] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:42.874588 24968 raft_consensus.cc:3037] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.874778 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fde63d6414745b0f81c0a0abe4f46" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:42.874972 24110 raft_consensus.cc:2454] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:42.875396 23781 leader_election.cc:302] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:42.875717 25058 raft_consensus.cc:2781] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:42.875854 25058 raft_consensus.cc:482] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:42.875954 25058 raft_consensus.cc:3037] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.878007 25134 tablet_bootstrap.cc:492] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:42.877977 24968 raft_consensus.cc:504] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.878768 25058 raft_consensus.cc:504] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:42.880308 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda032f2be744fe95170b3c3287d6ae" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:42.880318 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fde63d6414745b0f81c0a0abe4f46" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:42.880522 21228 raft_consensus.cc:3037] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.880798 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fde63d6414745b0f81c0a0abe4f46" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
W20240501 20:09:42.881467 23779 leader_election.cc:341] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:42.881677 23779 leader_election.cc:341] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:42.882643 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fde63d6414745b0f81c0a0abe4f46" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:42.882827 24110 raft_consensus.cc:3037] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.883982 21228 raft_consensus.cc:2454] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:42.884480 23647 leader_election.cc:302] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46; no voters:
I20240501 20:09:42.884763 25087 raft_consensus.cc:2781] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:42.884877 25087 raft_consensus.cc:686] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:42.885169 25087 consensus_queue.cc:237] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:42.885532 24110 raft_consensus.cc:2454] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:42.886102 24968 leader_election.cc:288] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:42.886087 23781 leader_election.cc:302] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:42.886580 25137 raft_consensus.cc:504] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.887692 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda032f2be744fe95170b3c3287d6ae" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:42.887760 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd04e795cbd4646beefd20cd7a38418" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:42.887959 23977 raft_consensus.cc:3037] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.888042 21357 raft_consensus.cc:3037] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.891119 21357 raft_consensus.cc:2454] T 7dd04e795cbd4646beefd20cd7a38418 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:42.891937 23910 leader_election.cc:302] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:42.892275 24948 raft_consensus.cc:2781] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:42.892450 24948 raft_consensus.cc:686] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:42.892727 24948 consensus_queue.cc:237] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:42.894026 23977 raft_consensus.cc:2454] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:42.894874 25137 leader_election.cc:288] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:42.895344 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd04e795cbd4646beefd20cd7a38418" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:42.895684 24110 raft_consensus.cc:3037] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.897091 25058 leader_election.cc:288] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:42.897478 25058 raft_consensus.cc:2781] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:42.897878 25058 raft_consensus.cc:686] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:42.899313 20824 catalog_manager.cc:5484] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:42.900297 20823 catalog_manager.cc:5484] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:42.901227 25126 tablet_bootstrap.cc:492] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:42.908125 25058 consensus_queue.cc:237] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:42.919957 24110 raft_consensus.cc:2454] T 7dd04e795cbd4646beefd20cd7a38418 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:42.923905 25107 tablet_bootstrap.cc:492] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:42.928133 25107 ts_tablet_manager.cc:1367] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.013s
I20240501 20:09:42.929497 25107 raft_consensus.cc:348] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:42.932921 25107 raft_consensus.cc:374] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240501 20:09:42.934381 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 48232us
I20240501 20:09:42.936169 25107 raft_consensus.cc:729] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.936642 25107 consensus_queue.cc:260] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:42.939877 25107 ts_tablet_manager.cc:1398] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:09:42.940201 25107 tablet_bootstrap.cc:492] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:42.940146 21357 raft_consensus.cc:1203] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:09:42.941884 25107 tablet_bootstrap.cc:654] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.947729 20823 catalog_manager.cc:5484] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:42.966154 23977 consensus_queue.cc:237] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:42.977731 20966 raft_consensus.cc:1261] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:09:42.978575 23910 consensus_peers.cc:485] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9fd7894fc5745ae9588437cd5bd0265. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:42.979210 23710 raft_consensus.cc:1261] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.980021 25107 tablet_bootstrap.cc:492] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:42.980201 25107 ts_tablet_manager.cc:1367] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.003s
I20240501 20:09:42.981082 25107 raft_consensus.cc:348] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:42.981313 25107 raft_consensus.cc:374] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.981400 25107 raft_consensus.cc:729] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.981813 25137 consensus_queue.cc:1035] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.982563 24948 consensus_queue.cc:1035] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:42.986208 25107 consensus_queue.cc:260] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:42.995730 25090 tablet_copy_client.cc:308] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:09:42.997099 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet ed53558d93214d34aedca8505870ec61 from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:45675)
I20240501 20:09:42.997303 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet ed53558d93214d34aedca8505870ec61 from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:45675: session id = 61380023001a4ee9ada9ec2c6c1b7159-ed53558d93214d34aedca8505870ec61
I20240501 20:09:42.998061 25126 tablet_bootstrap.cc:492] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:42.998534 25126 tablet_bootstrap.cc:492] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:09:42.998781 25126 ts_tablet_manager.cc:1367] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.021s
I20240501 20:09:42.999696 25126 raft_consensus.cc:348] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:42.981791 25129 tablet_copy_client.cc:308] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:09:43.000708 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 4587a9d5392742b58ec8d64619f2dda1 from peer 744c5a532e724391830c0785d32eb2fb ({username='slave'} at 127.13.251.135:46111)
I20240501 20:09:43.000854 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 4587a9d5392742b58ec8d64619f2dda1 from peer 744c5a532e724391830c0785d32eb2fb at {username='slave'} at 127.13.251.135:46111: session id = 744c5a532e724391830c0785d32eb2fb-4587a9d5392742b58ec8d64619f2dda1
I20240501 20:09:43.001428 23866 tablet_copy_source_session.cc:205] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.003187 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed53558d93214d34aedca8505870ec61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.003432 21378 tablet_copy_source_session.cc:205] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.004770 25129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4587a9d5392742b58ec8d64619f2dda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.009887 25107 ts_tablet_manager.cc:1398] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20240501 20:09:43.012274 25090 tablet_copy_client.cc:783] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:43.012501 25090 tablet_copy_client.cc:647] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:43.014669 25090 tablet_copy_client.cc:523] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.017881 25090 tablet_bootstrap.cc:492] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:43.017879 24948 raft_consensus.cc:2932] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:43.018644 25182 ts_tablet_manager.cc:902] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:43.020730 23711 raft_consensus.cc:2932] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:43.027545 20966 raft_consensus.cc:2932] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:43.031618 25126 raft_consensus.cc:729] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.031996 25126 consensus_queue.cc:260] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:43.032845 25126 ts_tablet_manager.cc:1398] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20240501 20:09:43.037694 20824 catalog_manager.cc:5484] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:43.040855 20987 tablet_copy_service.cc:342] P cda19d6a81c0473db302535ce2ada7f0: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-68133e033c824e11a1ed744bff76a934 received from {username='slave'} at 127.13.251.130:42109
I20240501 20:09:43.041026 20987 tablet_copy_service.cc:433] P cda19d6a81c0473db302535ce2ada7f0: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-68133e033c824e11a1ed744bff76a934 on tablet 68133e033c824e11a1ed744bff76a934 with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:09:43.056787 25129 tablet_copy_client.cc:783] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:43.057132 25129 tablet_copy_client.cc:647] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:43.061226 25129 tablet_copy_client.cc:523] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.061602 25090 tablet_bootstrap.cc:492] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:43.062033 25090 tablet_bootstrap.cc:492] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:09:43.062288 25090 ts_tablet_manager.cc:1367] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.012s
I20240501 20:09:43.063109 25090 raft_consensus.cc:348] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:43.063349 25090 raft_consensus.cc:729] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.065444 25129 tablet_bootstrap.cc:492] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:43.065886 25090 consensus_queue.cc:260] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:43.067070 25090 ts_tablet_manager.cc:1398] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:09:43.067906 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-ed53558d93214d34aedca8505870ec61 received from {username='slave'} at 127.13.251.132:45675
I20240501 20:09:43.068107 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-ed53558d93214d34aedca8505870ec61 on tablet ed53558d93214d34aedca8505870ec61 with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:09:43.073827 25182 tablet_copy_client.cc:308] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.133:37155
I20240501 20:09:43.075206 23734 tablet_copy_service.cc:140] P 51bf640e895b4e5eb91bdf7509ea7e46: Received BeginTabletCopySession request for tablet 4715a768081a49228ab47b8071c40c8f from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:50321)
I20240501 20:09:43.075433 23734 tablet_copy_service.cc:161] P 51bf640e895b4e5eb91bdf7509ea7e46: Beginning new tablet copy session on tablet 4715a768081a49228ab47b8071c40c8f from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:50321: session id = 4af096f5a6a9440a94f6682b399efe20-4715a768081a49228ab47b8071c40c8f
I20240501 20:09:43.079260 23734 tablet_copy_source_session.cc:205] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.081713 25182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4715a768081a49228ab47b8071c40c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.084025 25134 tablet_bootstrap.cc:492] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:43.084507 25134 tablet_bootstrap.cc:492] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:09:43.084803 25134 ts_tablet_manager.cc:1367] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.207s user 0.064s sys 0.018s
I20240501 20:09:43.085615 25134 raft_consensus.cc:348] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:43.085870 25134 raft_consensus.cc:729] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.086293 25134 consensus_queue.cc:260] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:43.086898 25134 ts_tablet_manager.cc:1398] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:09:43.087924 20987 tablet_copy_service.cc:342] P cda19d6a81c0473db302535ce2ada7f0: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-7a2f74307a194dc49d184d4ea288b36d received from {username='slave'} at 127.13.251.133:53673
I20240501 20:09:43.088069 20987 tablet_copy_service.cc:433] P cda19d6a81c0473db302535ce2ada7f0: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-7a2f74307a194dc49d184d4ea288b36d on tablet 7a2f74307a194dc49d184d4ea288b36d with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:09:43.101701 25182 tablet_copy_client.cc:783] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:43.102191 25182 tablet_copy_client.cc:647] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:43.104650 25182 tablet_copy_client.cc:523] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.112164 25182 tablet_bootstrap.cc:492] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:43.125944 25182 tablet_bootstrap.cc:492] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:43.126410 25182 tablet_bootstrap.cc:492] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:09:43.126719 25182 ts_tablet_manager.cc:1367] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.003s
I20240501 20:09:43.127720 25182 raft_consensus.cc:348] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:43.128044 25182 raft_consensus.cc:729] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.128489 25182 consensus_queue.cc:260] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:43.132757 25182 ts_tablet_manager.cc:1398] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:43.136543 23734 tablet_copy_service.cc:342] P 51bf640e895b4e5eb91bdf7509ea7e46: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-4715a768081a49228ab47b8071c40c8f received from {username='slave'} at 127.13.251.131:50321
I20240501 20:09:43.136672 23734 tablet_copy_service.cc:433] P 51bf640e895b4e5eb91bdf7509ea7e46: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-4715a768081a49228ab47b8071c40c8f on tablet 4715a768081a49228ab47b8071c40c8f with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:09:43.144147 25129 tablet_bootstrap.cc:492] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:43.144585 25129 tablet_bootstrap.cc:492] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: Bootstrap complete.
I20240501 20:09:43.144879 25129 ts_tablet_manager.cc:1367] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.080s user 0.041s sys 0.009s
I20240501 20:09:43.145567 25129 raft_consensus.cc:348] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.145813 25129 raft_consensus.cc:729] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.146065 25129 consensus_queue.cc:260] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.146649 25129 ts_tablet_manager.cc:1398] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:43.147607 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 744c5a532e724391830c0785d32eb2fb-4587a9d5392742b58ec8d64619f2dda1 received from {username='slave'} at 127.13.251.135:46111
I20240501 20:09:43.147958 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 744c5a532e724391830c0785d32eb2fb-4587a9d5392742b58ec8d64619f2dda1 on tablet 4587a9d5392742b58ec8d64619f2dda1 with peer 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:43.154280 21098 raft_consensus.cc:1203] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:09:43.161497 23845 consensus_queue.cc:237] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:43.163971 21098 raft_consensus.cc:1261] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:09:43.164074 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a23f60c87794a49ae1f82dff06fe881. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.164279 24110 raft_consensus.cc:1261] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:43.164496 25207 consensus_queue.cc:1035] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:43.164754 25058 consensus_queue.cc:1035] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:43.167258 25058 raft_consensus.cc:2932] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:43.167889 24110 raft_consensus.cc:2932] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:43.171111 21098 raft_consensus.cc:2932] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:43.173485 20824 catalog_manager.cc:5484] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:43.184710 24217 raft_consensus.cc:482] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:43.184871 24217 raft_consensus.cc:504] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:43.185598 24217 leader_election.cc:288] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:43.186267 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef0087e992043daa170b474d11a142b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:43.186388 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef0087e992043daa170b474d11a142b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:43.186533 23845 raft_consensus.cc:2454] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:43.186625 23977 raft_consensus.cc:2454] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:43.187057 24043 leader_election.cc:302] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:43.188269 24217 raft_consensus.cc:2781] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:43.188426 24217 raft_consensus.cc:482] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:43.188532 24217 raft_consensus.cc:3037] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.193312 24217 raft_consensus.cc:504] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:43.193977 24217 leader_election.cc:288] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:43.194306 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef0087e992043daa170b474d11a142b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:43.194360 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef0087e992043daa170b474d11a142b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:43.194499 23977 raft_consensus.cc:3037] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.194600 23845 raft_consensus.cc:3037] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.197335 23845 raft_consensus.cc:2454] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:43.199075 23977 raft_consensus.cc:2454] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:43.200567 24043 leader_election.cc:302] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:43.200937 24217 raft_consensus.cc:2781] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:43.201086 24217 raft_consensus.cc:686] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:43.201411 24217 consensus_queue.cc:237] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:43.204624 20823 catalog_manager.cc:5484] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:43.213531 21225 consensus_queue.cc:237] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:43.215797 21098 raft_consensus.cc:1261] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:09:43.215909 21164 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 -> Peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Couldn't send request to peer c6dcaa8efff642dcb7c2c6e8be3a65f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 420b46edf5fc438fb01e822440bce8f1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.216434 25074 consensus_queue.cc:1035] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:43.219305 25126 ts_tablet_manager.cc:902] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:43.224669 25126 tablet_copy_client.cc:308] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:09:43.225438 23977 raft_consensus.cc:1203] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:43.225611 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet 30eb2ed6c8b64204b4c3a11b41e53ec8 from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:58437)
I20240501 20:09:43.225764 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet 30eb2ed6c8b64204b4c3a11b41e53ec8 from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:58437: session id = feac73db0a3b4aafa374ca5dfd322b08-30eb2ed6c8b64204b4c3a11b41e53ec8
I20240501 20:09:43.227905 23997 tablet_copy_source_session.cc:205] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.230015 21357 raft_consensus.cc:1261] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:09:43.230304 25126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30eb2ed6c8b64204b4c3a11b41e53ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.230528 25230 consensus_queue.cc:1035] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:43.233307 25230 raft_consensus.cc:2932] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:43.234397 21357 raft_consensus.cc:2932] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:43.237004 25126 tablet_copy_client.cc:783] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:43.237350 25126 tablet_copy_client.cc:647] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:43.238880 23710 raft_consensus.cc:1203] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:43.239791 25126 tablet_copy_client.cc:523] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.240366 20823 catalog_manager.cc:5484] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:43.247288 25126 tablet_bootstrap.cc:492] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:43.254936 21098 raft_consensus.cc:2932] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:43.256273 25126 tablet_bootstrap.cc:492] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:43.256708 25126 tablet_bootstrap.cc:492] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:09:43.256953 25126 ts_tablet_manager.cc:1367] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 20:09:43.257550 25126 raft_consensus.cc:348] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:43.257819 25126 raft_consensus.cc:729] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.260046 25126 consensus_queue.cc:260] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:43.264818 25126 ts_tablet_manager.cc:1398] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:09:43.265483 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-30eb2ed6c8b64204b4c3a11b41e53ec8 received from {username='slave'} at 127.13.251.130:58437
I20240501 20:09:43.265631 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-30eb2ed6c8b64204b4c3a11b41e53ec8 on tablet 30eb2ed6c8b64204b4c3a11b41e53ec8 with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:09:43.269142 25087 consensus_queue.cc:1035] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.272218 24948 raft_consensus.cc:482] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:43.272408 24948 raft_consensus.cc:504] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:43.273102 24948 leader_election.cc:288] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:43.274394 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fde63d6414745b0f81c0a0abe4f46" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:43.275014 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37fde63d6414745b0f81c0a0abe4f46" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:43.275354 24110 raft_consensus.cc:2379] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 744c5a532e724391830c0785d32eb2fb in current term 1: Already voted for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in this term.
I20240501 20:09:43.276522 23911 leader_election.cc:302] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb; no voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:43.278028 25087 consensus_queue.cc:1035] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.278654 24948 raft_consensus.cc:3037] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.282121 24948 raft_consensus.cc:2726] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:43.283751 21357 raft_consensus.cc:1203] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:09:43.311479 25058 consensus_queue.cc:1035] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.319676 24948 consensus_queue.cc:1035] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.323812 24948 consensus_queue.cc:1035] T 7dd04e795cbd4646beefd20cd7a38418 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.330055 25058 consensus_queue.cc:1035] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.335974 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:43.344017 23845 raft_consensus.cc:1261] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:43.345444 24217 consensus_queue.cc:1035] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.353495 20823 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 5ef0087e992043daa170b474d11a142b (table rebalancer_dynamic_table [id=70d6282b175e476aa7efeb98d8e6c7fb]). Expected version 1 got 0
I20240501 20:09:43.357640 25267 tablet.cc:1722] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:43.358016 23977 raft_consensus.cc:1261] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:43.358794 25217 consensus_queue.cc:1035] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.361840 25267 tablet.cc:1620] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:43.366528 20823 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 5ef0087e992043daa170b474d11a142b (table rebalancer_dynamic_table [id=70d6282b175e476aa7efeb98d8e6c7fb]). Expected version 1 got 0
I20240501 20:09:43.369539 25090 ts_tablet_manager.cc:902] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:43.375909 21225 raft_consensus.cc:1203] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:09:43.385138 20824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=70d6282b175e476aa7efeb98d8e6c7fb] alter complete (version 1)
I20240501 20:09:43.389619 20824 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet 5ef0087e992043daa170b474d11a142b (table rebalancer_dynamic_table [id=70d6282b175e476aa7efeb98d8e6c7fb]). Expected version 1 got 0
I20240501 20:09:43.386938 25272 tablet.cc:1722] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:43.393536 25272 tablet.cc:1620] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:43.393800 20824 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet 5ef0087e992043daa170b474d11a142b (table rebalancer_dynamic_table [id=70d6282b175e476aa7efeb98d8e6c7fb]). Expected version 1 got 0
I20240501 20:09:43.398093 25265 tablet.cc:1722] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:43.398873 25265 tablet.cc:1620] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:43.409826 25090 tablet_copy_client.cc:308] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:09:43.410954 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet a9fd7894fc5745ae9588437cd5bd0265 from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:41549)
I20240501 20:09:43.411108 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet a9fd7894fc5745ae9588437cd5bd0265 from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:41549: session id = 61380023001a4ee9ada9ec2c6c1b7159-a9fd7894fc5745ae9588437cd5bd0265
I20240501 20:09:43.413420 23997 tablet_copy_source_session.cc:205] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.414836 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fd7894fc5745ae9588437cd5bd0265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.419420 25090 tablet_copy_client.cc:783] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:43.419669 25090 tablet_copy_client.cc:647] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:43.421511 25090 tablet_copy_client.cc:523] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.424659 25090 tablet_bootstrap.cc:492] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:43.431354 25090 tablet_bootstrap.cc:492] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:43.431829 25090 tablet_bootstrap.cc:492] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:09:43.432093 25090 ts_tablet_manager.cc:1367] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20240501 20:09:43.432930 25090 raft_consensus.cc:348] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:43.433243 25090 raft_consensus.cc:729] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.433799 25090 consensus_queue.cc:260] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:43.435596 25090 ts_tablet_manager.cc:1398] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:43.437216 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-a9fd7894fc5745ae9588437cd5bd0265 received from {username='slave'} at 127.13.251.132:41549
I20240501 20:09:43.437356 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-a9fd7894fc5745ae9588437cd5bd0265 on tablet a9fd7894fc5745ae9588437cd5bd0265 with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:09:43.467892 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:43.483939 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 88f0ef64f05640c1a9e58e528149f8d0 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.484436 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f0ef64f05640c1a9e58e528149f8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.485340 23955 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3951bb18524e41aa6836122e97e7f5 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.485761 23955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3951bb18524e41aa6836122e97e7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.485677 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 88f0ef64f05640c1a9e58e528149f8d0 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.486109 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f0ef64f05640c1a9e58e528149f8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.486423 23956 tablet_service.cc:1467] Processing CreateTablet for tablet b7afaa4052754794b8f220501887af08 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.486804 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7afaa4052754794b8f220501887af08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.487975 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 88f0ef64f05640c1a9e58e528149f8d0 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.488373 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f0ef64f05640c1a9e58e528149f8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.491402 20824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:43.492005 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3951bb18524e41aa6836122e97e7f5 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.492461 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3951bb18524e41aa6836122e97e7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.493551 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3951bb18524e41aa6836122e97e7f5 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.494659 23824 tablet_service.cc:1467] Processing CreateTablet for tablet b7afaa4052754794b8f220501887af08 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.495069 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7afaa4052754794b8f220501887af08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.499733 25107 tablet_bootstrap.cc:492] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:43.501238 25107 tablet_bootstrap.cc:654] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.503437 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3951bb18524e41aa6836122e97e7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.505350 24090 tablet_service.cc:1467] Processing CreateTablet for tablet b7afaa4052754794b8f220501887af08 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.505833 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7afaa4052754794b8f220501887af08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.519596 25265 tablet.cc:1722] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:43.520126 25265 tablet.cc:1620] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:43.521636 25267 tablet.cc:1722] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:43.522120 25267 tablet.cc:1620] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:43.523902 25272 tablet.cc:1722] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:43.524354 25272 tablet.cc:1620] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:43.534463 20824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=70d6282b175e476aa7efeb98d8e6c7fb] alter complete (version 2)
I20240501 20:09:43.539119 25304 tablet_bootstrap.cc:492] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:43.549700 25302 tablet_bootstrap.cc:492] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:43.549778 25304 tablet_bootstrap.cc:654] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.553124 25302 tablet_bootstrap.cc:654] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.554785 25303 tablet_bootstrap.cc:492] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:43.555884 25107 tablet_bootstrap.cc:492] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:43.556116 25107 ts_tablet_manager.cc:1367] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.009s
I20240501 20:09:43.556761 25303 tablet_bootstrap.cc:654] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.556926 25107 raft_consensus.cc:348] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.557214 25107 raft_consensus.cc:374] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.557333 25107 raft_consensus.cc:729] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.557665 25107 consensus_queue.cc:260] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.559314 25107 ts_tablet_manager.cc:1398] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:43.560696 25107 tablet_bootstrap.cc:492] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:43.567006 25303 tablet_bootstrap.cc:492] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:43.567245 25303 ts_tablet_manager.cc:1367] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240501 20:09:43.568148 25303 raft_consensus.cc:348] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.568280 25302 tablet_bootstrap.cc:492] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:43.568432 25302 ts_tablet_manager.cc:1367] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240501 20:09:43.568437 25303 raft_consensus.cc:374] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.568536 25303 raft_consensus.cc:729] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.568845 25303 consensus_queue.cc:260] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.569604 25303 ts_tablet_manager.cc:1398] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:43.569929 25303 tablet_bootstrap.cc:492] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:43.570928 25107 tablet_bootstrap.cc:654] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.571846 25303 tablet_bootstrap.cc:654] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.573477 25304 tablet_bootstrap.cc:492] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:43.573738 25304 ts_tablet_manager.cc:1367] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240501 20:09:43.574235 25302 raft_consensus.cc:348] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.574496 25302 raft_consensus.cc:374] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.574604 25302 raft_consensus.cc:729] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.574983 25302 consensus_queue.cc:260] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.575690 25302 ts_tablet_manager.cc:1398] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:43.582603 25303 tablet_bootstrap.cc:492] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:43.576747 25304 raft_consensus.cc:348] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.583083 25303 ts_tablet_manager.cc:1367] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20240501 20:09:43.584007 25303 raft_consensus.cc:348] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:43.584370 25303 raft_consensus.cc:374] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.583113 25304 raft_consensus.cc:374] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.584501 25303 raft_consensus.cc:729] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.584753 25303 consensus_queue.cc:260] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:43.585569 25107 tablet_bootstrap.cc:492] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:43.585575 25304 raft_consensus.cc:729] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.585788 25107 ts_tablet_manager.cc:1367] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.002s
I20240501 20:09:43.586177 25303 ts_tablet_manager.cc:1398] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:43.586084 25304 consensus_queue.cc:260] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.586468 25303 tablet_bootstrap.cc:492] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:43.586753 25107 raft_consensus.cc:348] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:43.587184 25107 raft_consensus.cc:374] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.587302 25107 raft_consensus.cc:729] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.587652 25107 consensus_queue.cc:260] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:43.588990 25303 tablet_bootstrap.cc:654] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.589340 25107 ts_tablet_manager.cc:1398] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:43.589699 25107 tablet_bootstrap.cc:492] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:43.593384 25304 ts_tablet_manager.cc:1398] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240501 20:09:43.595053 25107 tablet_bootstrap.cc:654] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.597883 25303 tablet_bootstrap.cc:492] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:43.598063 25303 ts_tablet_manager.cc:1367] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240501 20:09:43.598496 25304 tablet_bootstrap.cc:492] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:43.598863 25303 raft_consensus.cc:348] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.599074 25303 raft_consensus.cc:374] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.599177 25303 raft_consensus.cc:729] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.599462 25303 consensus_queue.cc:260] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.600811 25303 ts_tablet_manager.cc:1398] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:43.602286 25107 tablet_bootstrap.cc:492] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:43.602438 25107 ts_tablet_manager.cc:1367] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20240501 20:09:43.603865 25304 tablet_bootstrap.cc:654] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.604068 25107 raft_consensus.cc:348] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.604285 25107 raft_consensus.cc:374] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.604382 25107 raft_consensus.cc:729] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.604604 25107 consensus_queue.cc:260] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.605177 25107 ts_tablet_manager.cc:1398] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240501 20:09:43.627947 25304 tablet_bootstrap.cc:492] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:43.628546 25304 ts_tablet_manager.cc:1367] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.012s
I20240501 20:09:43.629714 25304 raft_consensus.cc:348] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:43.630218 25304 raft_consensus.cc:374] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.630367 25304 raft_consensus.cc:729] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.630738 25304 consensus_queue.cc:260] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:43.632887 25304 ts_tablet_manager.cc:1398] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:09:43.637049 20825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:43.645519 25265 tablet.cc:1722] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:43.646333 25267 tablet.cc:1722] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:43.647550 25272 tablet.cc:1722] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:43.654474 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=70d6282b175e476aa7efeb98d8e6c7fb] alter complete (version 3)
I20240501 20:09:43.674412 21208 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:43.728381 24948 raft_consensus.cc:482] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:43.728611 24948 raft_consensus.cc:504] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.729544 24948 leader_election.cc:288] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:43.731751 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f0ef64f05640c1a9e58e528149f8d0" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:43.732081 23845 raft_consensus.cc:2454] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:43.733457 23912 leader_election.cc:302] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:43.734109 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f0ef64f05640c1a9e58e528149f8d0" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:43.734400 24110 raft_consensus.cc:2454] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:43.735253 24948 raft_consensus.cc:2781] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:43.735457 24948 raft_consensus.cc:482] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:43.735616 24948 raft_consensus.cc:3037] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.736495 25217 raft_consensus.cc:482] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:43.736654 25217 raft_consensus.cc:504] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.737471 25217 leader_election.cc:288] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:43.738482 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3951bb18524e41aa6836122e97e7f5" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:43.738556 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3951bb18524e41aa6836122e97e7f5" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:43.738740 20967 raft_consensus.cc:2454] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:43.738797 23977 raft_consensus.cc:2454] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:43.739342 24043 leader_election.cc:302] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:43.740017 25217 raft_consensus.cc:2781] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:43.740135 25217 raft_consensus.cc:482] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:43.740264 25217 raft_consensus.cc:3037] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.748127 24948 raft_consensus.cc:504] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.749652 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f0ef64f05640c1a9e58e528149f8d0" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:43.749944 23845 raft_consensus.cc:3037] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.753969 23845 raft_consensus.cc:2454] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:43.755014 23912 leader_election.cc:302] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:43.755437 25138 raft_consensus.cc:2781] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:43.755630 25138 raft_consensus.cc:686] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:43.755991 25138 consensus_queue.cc:237] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.757213 20825 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:43.757478 20825 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:43.760890 20825 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 5ef0087e992043daa170b474d11a142b
I20240501 20:09:43.761741 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef0087e992043daa170b474d11a142b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:43 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:43.762863 23956 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef0087e992043daa170b474d11a142b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:43 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:43.763567 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef0087e992043daa170b474d11a142b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:43 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:43.768191 20825 catalog_manager.cc:5484] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:43.769431 25217 raft_consensus.cc:504] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.775657 25217 leader_election.cc:288] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:43.776113 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3951bb18524e41aa6836122e97e7f5" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:43.776371 20967 raft_consensus.cc:3037] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.779234 24948 leader_election.cc:288] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:43.779266 20967 raft_consensus.cc:2454] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:43.779316 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f0ef64f05640c1a9e58e528149f8d0" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:43.779484 24110 raft_consensus.cc:3037] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.781062 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3951bb18524e41aa6836122e97e7f5" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:43.781314 23977 raft_consensus.cc:3037] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.782852 24110 raft_consensus.cc:2454] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:43.784101 23977 raft_consensus.cc:2454] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:43.784610 24043 leader_election.cc:302] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:43.786003 25217 raft_consensus.cc:2781] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:43.786692 25217 raft_consensus.cc:686] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:43.787065 25217 consensus_queue.cc:237] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:43.791417 21336 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:09:43.792586 21098 raft_consensus.cc:1203] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:09:43.798795 25327 tablet_replica.cc:330] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:43.799017 25327 raft_consensus.cc:2229] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:43.799228 25327 raft_consensus.cc:2258] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:43.800346 25327 ts_tablet_manager.cc:1875] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:43.800782 25323 ts_tablet_manager.cc:902] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:43.802037 25324 ts_tablet_manager.cc:902] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:43.803908 25328 tablet_replica.cc:330] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:43.804564 25328 raft_consensus.cc:2229] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:43.805053 25328 raft_consensus.cc:2258] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:43.806375 25328 ts_tablet_manager.cc:1875] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:43.807524 20822 catalog_manager.cc:5484] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:43.826359 23956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:43.828166 25326 tablet_replica.cc:330] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:43.828747 23689 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:43.831975 25326 raft_consensus.cc:2229] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:43.834089 25326 raft_consensus.cc:2258] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:43.836258 25327 ts_tablet_manager.cc:1888] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:43.837241 25326 ts_tablet_manager.cc:1875] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:43.841051 25327 log.cc:1196] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/5ef0087e992043daa170b474d11a142b
I20240501 20:09:43.848235 21358 raft_consensus.cc:1203] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:09:43.848464 25327 ts_tablet_manager.cc:1909] T 5ef0087e992043daa170b474d11a142b P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:43.849807 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 5ef0087e992043daa170b474d11a142b (table rebalancer_dynamic_table_new_name [id=70d6282b175e476aa7efeb98d8e6c7fb]) successfully deleted
I20240501 20:09:43.860409 25323 tablet_copy_client.cc:308] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:09:43.863962 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet 7a23f60c87794a49ae1f82dff06fe881 from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:54857)
I20240501 20:09:43.864154 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet 7a23f60c87794a49ae1f82dff06fe881 from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:54857: session id = 4af096f5a6a9440a94f6682b399efe20-7a23f60c87794a49ae1f82dff06fe881
I20240501 20:09:43.865224 20822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:43.867262 23866 tablet_copy_source_session.cc:205] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.868404 25328 ts_tablet_manager.cc:1888] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:43.869668 25328 log.cc:1196] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/5ef0087e992043daa170b474d11a142b
I20240501 20:09:43.870250 25328 ts_tablet_manager.cc:1909] T 5ef0087e992043daa170b474d11a142b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:09:43.871448 23825 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:43.872529 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 5ef0087e992043daa170b474d11a142b (table rebalancer_dynamic_table_new_name [id=70d6282b175e476aa7efeb98d8e6c7fb]) successfully deleted
I20240501 20:09:43.874924 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 46b86f450dbe4db4bbec815b59a959f2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6683ee4f249143d1adbebbee7d1db0e6]), partition=
I20240501 20:09:43.875211 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 46b86f450dbe4db4bbec815b59a959f2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6683ee4f249143d1adbebbee7d1db0e6]), partition=
I20240501 20:09:43.875316 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b86f450dbe4db4bbec815b59a959f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.875568 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b86f450dbe4db4bbec815b59a959f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.881920 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 46b86f450dbe4db4bbec815b59a959f2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6683ee4f249143d1adbebbee7d1db0e6]), partition=
I20240501 20:09:43.882268 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b86f450dbe4db4bbec815b59a959f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.893472 20945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:43.896514 21077 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:43.908756 25323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a23f60c87794a49ae1f82dff06fe881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.910571 25304 tablet_bootstrap.cc:492] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:43.912369 25304 tablet_bootstrap.cc:654] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.913889 24088 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:43.914703 25303 tablet_bootstrap.cc:492] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:43.916772 25303 tablet_bootstrap.cc:654] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.922068 25326 ts_tablet_manager.cc:1888] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:43.927472 25326 log.cc:1196] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/5ef0087e992043daa170b474d11a142b
I20240501 20:09:43.928002 25326 ts_tablet_manager.cc:1909] T 5ef0087e992043daa170b474d11a142b P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:43.931195 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 5ef0087e992043daa170b474d11a142b (table rebalancer_dynamic_table_new_name [id=70d6282b175e476aa7efeb98d8e6c7fb]) successfully deleted
I20240501 20:09:43.938225 25304 tablet_bootstrap.cc:492] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:43.938387 25304 ts_tablet_manager.cc:1367] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20240501 20:09:43.939123 25304 raft_consensus.cc:348] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:43.939342 25304 raft_consensus.cc:374] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.939437 25304 raft_consensus.cc:729] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.939697 25304 consensus_queue.cc:260] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:43.940253 25304 ts_tablet_manager.cc:1398] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:43.940894 25345 tablet_bootstrap.cc:492] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:43.942545 25345 tablet_bootstrap.cc:654] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:43.946733 25303 tablet_bootstrap.cc:492] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:43.946949 25303 ts_tablet_manager.cc:1367] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.006s
I20240501 20:09:43.947722 25303 raft_consensus.cc:348] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:43.947952 25303 raft_consensus.cc:374] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:43.948026 25303 raft_consensus.cc:729] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:43.948267 25303 consensus_queue.cc:260] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:43.949847 25303 ts_tablet_manager.cc:1398] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:43.962124 25324 tablet_copy_client.cc:308] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:09:43.963016 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet 420b46edf5fc438fb01e822440bce8f1 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 ({username='slave'} at 127.13.251.134:60949)
I20240501 20:09:43.963194 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet 420b46edf5fc438fb01e822440bce8f1 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 at {username='slave'} at 127.13.251.134:60949: session id = c6dcaa8efff642dcb7c2c6e8be3a65f7-420b46edf5fc438fb01e822440bce8f1
I20240501 20:09:43.964759 25323 tablet_copy_client.cc:783] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:43.965018 25323 tablet_copy_client.cc:647] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:43.966221 21248 tablet_copy_source_session.cc:205] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:43.969226 25323 tablet_copy_client.cc:523] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:43.972525 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420b46edf5fc438fb01e822440bce8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.985685 25323 tablet_bootstrap.cc:492] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:44.013998 25323 tablet_bootstrap.cc:492] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:44.014514 25323 tablet_bootstrap.cc:492] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:09:44.014794 25323 ts_tablet_manager.cc:1367] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20240501 20:09:44.015578 25323 raft_consensus.cc:348] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:44.015918 25323 raft_consensus.cc:729] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.016227 25323 consensus_queue.cc:260] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:44.016952 25323 ts_tablet_manager.cc:1398] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:44.017776 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-7a23f60c87794a49ae1f82dff06fe881 received from {username='slave'} at 127.13.251.131:54857
I20240501 20:09:44.017892 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-7a23f60c87794a49ae1f82dff06fe881 on tablet 7a23f60c87794a49ae1f82dff06fe881 with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:09:44.020004 25324 tablet_copy_client.cc:783] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.020802 25324 tablet_copy_client.cc:647] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.023109 25345 tablet_bootstrap.cc:492] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:44.023329 25345 ts_tablet_manager.cc:1367] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.000s
I20240501 20:09:44.024216 25345 raft_consensus.cc:348] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:44.024518 25345 raft_consensus.cc:374] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.024654 25345 raft_consensus.cc:729] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.024993 25345 consensus_queue.cc:260] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:44.025859 25345 ts_tablet_manager.cc:1398] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:44.031018 25324 tablet_copy_client.cc:523] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:44.042181 25217 raft_consensus.cc:482] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:44.042356 25217 raft_consensus.cc:504] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:44.043031 25217 leader_election.cc:288] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:44.043771 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7afaa4052754794b8f220501887af08" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:44.044003 23844 raft_consensus.cc:2454] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:44.044562 24046 leader_election.cc:302] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:44.044572 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7afaa4052754794b8f220501887af08" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:44.044786 23977 raft_consensus.cc:2454] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:44.044929 25217 raft_consensus.cc:2781] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:44.045032 25217 raft_consensus.cc:482] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:44.045112 25217 raft_consensus.cc:3037] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.047616 25324 tablet_bootstrap.cc:492] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:44.047920 25217 raft_consensus.cc:504] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:44.050036 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7afaa4052754794b8f220501887af08" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:44.050246 23844 raft_consensus.cc:3037] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.052482 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7afaa4052754794b8f220501887af08" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:44.052704 23977 raft_consensus.cc:3037] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.053126 23844 raft_consensus.cc:2454] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:44.053560 24046 leader_election.cc:302] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:44.053881 24217 raft_consensus.cc:2781] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:44.055346 23977 raft_consensus.cc:2454] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:44.055971 25217 leader_election.cc:288] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:44.072667 24217 raft_consensus.cc:686] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:44.073064 24217 consensus_queue.cc:237] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:44.107689 20822 catalog_manager.cc:5484] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:44.136276 25324 tablet_bootstrap.cc:492] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:44.136790 25324 tablet_bootstrap.cc:492] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap complete.
I20240501 20:09:44.137060 25324 ts_tablet_manager.cc:1367] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.090s user 0.056s sys 0.000s
I20240501 20:09:44.137702 25324 raft_consensus.cc:348] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:44.137930 25324 raft_consensus.cc:729] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.138187 25324 consensus_queue.cc:260] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:44.138921 25324 ts_tablet_manager.cc:1398] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:44.139688 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-420b46edf5fc438fb01e822440bce8f1 received from {username='slave'} at 127.13.251.134:60949
I20240501 20:09:44.139847 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-420b46edf5fc438fb01e822440bce8f1 on tablet 420b46edf5fc438fb01e822440bce8f1 with peer c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:44.192467 25350 raft_consensus.cc:482] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:44.192653 25350 raft_consensus.cc:504] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:44.193328 25350 leader_election.cc:288] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:44.193841 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b86f450dbe4db4bbec815b59a959f2" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:44.194125 23844 raft_consensus.cc:2454] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:44.194620 24046 leader_election.cc:302] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:44.194953 25350 raft_consensus.cc:2781] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:44.195118 25350 raft_consensus.cc:482] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:44.195230 25350 raft_consensus.cc:3037] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.198071 25350 raft_consensus.cc:504] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:44.200474 21228 raft_consensus.cc:1203] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:09:44.203166 24948 consensus_queue.cc:1035] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.204121 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b86f450dbe4db4bbec815b59a959f2" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:44.204360 21098 raft_consensus.cc:2454] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:44.206773 25350 leader_election.cc:288] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:44.207201 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b86f450dbe4db4bbec815b59a959f2" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:44.207445 21098 raft_consensus.cc:3037] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.208463 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b86f450dbe4db4bbec815b59a959f2" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:44.208659 23845 raft_consensus.cc:3037] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.210239 21098 raft_consensus.cc:2454] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:44.210649 24043 leader_election.cc:302] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:44.210961 25350 raft_consensus.cc:2781] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:44.211048 23845 raft_consensus.cc:2454] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:44.211282 25350 raft_consensus.cc:686] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:44.211781 25350 consensus_queue.cc:237] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:44.227120 23845 raft_consensus.cc:1203] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:44.245638 25138 consensus_queue.cc:1035] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.247231 20824 catalog_manager.cc:5484] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:44.327593 25350 consensus_queue.cc:1035] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.339227 25217 consensus_queue.cc:1035] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.343380 23956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:44.343616 23689 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:44.369524 21208 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:44.370182 21336 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:09:44.373243 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:44.381572 23844 raft_consensus.cc:1261] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:44.383085 24217 consensus_queue.cc:1035] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.384718 21098 raft_consensus.cc:1261] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:44.387238 24217 consensus_queue.cc:1035] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.409886 20823 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 46b86f450dbe4db4bbec815b59a959f2 (table rebalancer_dynamic_table [id=6683ee4f249143d1adbebbee7d1db0e6]). Expected version 1 got 0
I20240501 20:09:44.453258 20823 catalog_manager.cc:5576] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271) does not have the latest schema for tablet 46b86f450dbe4db4bbec815b59a959f2 (table rebalancer_dynamic_table [id=6683ee4f249143d1adbebbee7d1db0e6]). Expected version 1 got 0
I20240501 20:09:44.464315 25416 tablet.cc:1722] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:44.464826 25416 tablet.cc:1620] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:44.465263 25417 tablet.cc:1722] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:44.467532 25417 tablet.cc:1620] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:44.467756 20823 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 46b86f450dbe4db4bbec815b59a959f2 (table rebalancer_dynamic_table [id=6683ee4f249143d1adbebbee7d1db0e6]). Expected version 1 got 0
I20240501 20:09:44.471000 24089 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:44.473407 25419 tablet.cc:1722] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:44.477564 20823 catalog_manager.cc:5576] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271) does not have the latest schema for tablet 46b86f450dbe4db4bbec815b59a959f2 (table rebalancer_dynamic_table [id=6683ee4f249143d1adbebbee7d1db0e6]). Expected version 1 got 0
I20240501 20:09:44.480670 25419 tablet.cc:1620] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: MemRowSet was empty: no flush needed.
I20240501 20:09:44.481051 23825 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:44.491408 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=6683ee4f249143d1adbebbee7d1db0e6] alter complete (version 1)
I20240501 20:09:44.508926 21076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:44.527240 20945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:44.546237 24217 consensus_queue.cc:1035] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.581503 25217 consensus_queue.cc:1035] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.600190 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:44.613732 25416 tablet.cc:1722] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:44.614264 25416 tablet.cc:1620] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:44.616215 25419 tablet.cc:1722] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:44.616685 25419 tablet.cc:1620] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: MemRowSet was empty: no flush needed.
I20240501 20:09:44.619591 20818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:44.641108 23956 tablet_service.cc:1467] Processing CreateTablet for tablet cf1ac1f2f3d642e9a9c785bb4963af1f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.641594 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1ac1f2f3d642e9a9c785bb4963af1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.642357 21336 tablet_service.cc:1467] Processing CreateTablet for tablet fcbdc6727cab4983984d49882868014e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.642496 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 25dee912ab5f4613b66d056a8731f188 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.642817 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbdc6727cab4983984d49882868014e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.642963 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25dee912ab5f4613b66d056a8731f188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.644301 23825 tablet_service.cc:1467] Processing CreateTablet for tablet cf1ac1f2f3d642e9a9c785bb4963af1f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.644687 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1ac1f2f3d642e9a9c785bb4963af1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.648914 20945 tablet_service.cc:1467] Processing CreateTablet for tablet fcbdc6727cab4983984d49882868014e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.649371 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbdc6727cab4983984d49882868014e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.651039 23689 tablet_service.cc:1467] Processing CreateTablet for tablet cf1ac1f2f3d642e9a9c785bb4963af1f (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.651451 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1ac1f2f3d642e9a9c785bb4963af1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.652734 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=6683ee4f249143d1adbebbee7d1db0e6] alter complete (version 2)
I20240501 20:09:44.655941 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 25dee912ab5f4613b66d056a8731f188 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.656306 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25dee912ab5f4613b66d056a8731f188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.663419 25417 tablet.cc:1722] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:44.663991 25417 tablet.cc:1620] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:44.673101 23955 tablet_service.cc:1467] Processing CreateTablet for tablet 25dee912ab5f4613b66d056a8731f188 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.673569 23955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25dee912ab5f4613b66d056a8731f188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.680729 23957 tablet_service.cc:1467] Processing CreateTablet for tablet fcbdc6727cab4983984d49882868014e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.681121 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbdc6727cab4983984d49882868014e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.775560 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:44.789924 25416 tablet.cc:1722] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:44.790788 25417 tablet.cc:1722] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:44.794718 25419 tablet.cc:1722] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:44.803647 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=6683ee4f249143d1adbebbee7d1db0e6] alter complete (version 3)
I20240501 20:09:44.827476 25431 tablet_bootstrap.cc:492] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:44.830510 25431 tablet_bootstrap.cc:654] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.831938 25430 tablet_bootstrap.cc:492] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:44.838224 25430 tablet_bootstrap.cc:654] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
W20240501 20:09:44.838636 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 174318us
I20240501 20:09:44.868386 25427 tablet_bootstrap.cc:492] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:44.870633 25427 tablet_bootstrap.cc:654] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.876840 25430 tablet_bootstrap.cc:492] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:44.877033 25431 tablet_bootstrap.cc:492] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:44.877122 25430 ts_tablet_manager.cc:1367] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20240501 20:09:44.877365 25431 ts_tablet_manager.cc:1367] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20240501 20:09:44.878211 25431 raft_consensus.cc:348] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.878616 25431 raft_consensus.cc:374] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.878763 25431 raft_consensus.cc:729] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.878837 25430 raft_consensus.cc:348] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.879158 25430 raft_consensus.cc:374] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.879324 25430 raft_consensus.cc:729] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.879235 25431 consensus_queue.cc:260] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.879736 25430 consensus_queue.cc:260] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.880584 25430 ts_tablet_manager.cc:1398] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:44.880882 25430 tablet_bootstrap.cc:492] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:44.883127 25430 tablet_bootstrap.cc:654] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.886293 25431 ts_tablet_manager.cc:1398] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:09:44.892957 25426 tablet_bootstrap.cc:492] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:44.896595 25425 tablet_bootstrap.cc:492] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:44.896925 25431 tablet_bootstrap.cc:492] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:44.905436 25431 tablet_bootstrap.cc:654] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.906162 25430 tablet_bootstrap.cc:492] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:44.906414 25426 tablet_bootstrap.cc:654] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.906363 25430 ts_tablet_manager.cc:1367] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.020s
I20240501 20:09:44.907388 25427 tablet_bootstrap.cc:492] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:44.907632 25427 ts_tablet_manager.cc:1367] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.004s
I20240501 20:09:44.907518 25430 raft_consensus.cc:348] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:44.907753 25430 raft_consensus.cc:374] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.907853 25430 raft_consensus.cc:729] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.908147 25430 consensus_queue.cc:260] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:44.908748 25430 ts_tablet_manager.cc:1398] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:44.908844 25427 raft_consensus.cc:348] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.909081 25427 raft_consensus.cc:374] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.909188 25427 raft_consensus.cc:729] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.909444 25427 consensus_queue.cc:260] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.910087 25427 ts_tablet_manager.cc:1398] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:44.913776 20818 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:44.914011 20818 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:44.917582 20818 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 46b86f450dbe4db4bbec815b59a959f2
I20240501 20:09:44.918565 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 46b86f450dbe4db4bbec815b59a959f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:44 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:44.919113 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet 46b86f450dbe4db4bbec815b59a959f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:44 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:44.920094 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 46b86f450dbe4db4bbec815b59a959f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:44 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:44.926005 25425 tablet_bootstrap.cc:654] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.943102 25431 tablet_bootstrap.cc:492] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:44.944245 25425 tablet_bootstrap.cc:492] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:44.944487 25425 ts_tablet_manager.cc:1367] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.000s
I20240501 20:09:44.945461 25425 raft_consensus.cc:348] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.946213 25442 tablet_replica.cc:330] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:44.946193 25431 ts_tablet_manager.cc:1367] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.006s
I20240501 20:09:44.946411 25441 tablet_replica.cc:330] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:44.947216 25431 raft_consensus.cc:348] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:44.948060 25425 raft_consensus.cc:374] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.948163 25440 tablet_replica.cc:330] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:44.948351 25425 raft_consensus.cc:729] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.948400 25441 raft_consensus.cc:2229] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:44.948518 25442 raft_consensus.cc:2229] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:44.948632 25441 raft_consensus.cc:2258] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:44.948949 25442 raft_consensus.cc:2258] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:44.949236 25440 raft_consensus.cc:2229] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:44.949134 25425 consensus_queue.cc:260] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.949585 25431 raft_consensus.cc:374] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.949744 25428 raft_consensus.cc:482] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:44.949784 25441 ts_tablet_manager.cc:1875] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:44.949900 25428 raft_consensus.cc:504] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.950682 25428 leader_election.cc:288] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:44.951220 25431 raft_consensus.cc:729] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.951732 25425 ts_tablet_manager.cc:1398] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:09:44.951337 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbdc6727cab4983984d49882868014e" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
W20240501 20:09:44.953649 20900 leader_election.cc:341] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:44.954315 25442 ts_tablet_manager.cc:1875] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:44.954795 25440 raft_consensus.cc:2258] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:44.954926 25431 consensus_queue.cc:260] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:44.951984 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbdc6727cab4983984d49882868014e" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:44.955842 21358 raft_consensus.cc:2454] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:44.956364 20901 leader_election.cc:302] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters: 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:44.956619 25428 raft_consensus.cc:2781] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:44.956722 25428 raft_consensus.cc:482] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:44.956813 25428 raft_consensus.cc:3037] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.959470 25428 raft_consensus.cc:504] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.962900 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbdc6727cab4983984d49882868014e" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:44.963196 21358 raft_consensus.cc:3037] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.966688 21358 raft_consensus.cc:2454] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:44.967592 20901 leader_election.cc:302] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:44.967957 25148 raft_consensus.cc:2781] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:44.968098 25148 raft_consensus.cc:686] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:44.968465 25148 consensus_queue.cc:237] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:44.969408 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbdc6727cab4983984d49882868014e" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:44.968487 25428 leader_election.cc:288] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
W20240501 20:09:44.972344 20900 leader_election.cc:341] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:44.978018 25440 ts_tablet_manager.cc:1875] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:44.987947 25441 ts_tablet_manager.cc:1888] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:44.988081 25441 log.cc:1196] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/46b86f450dbe4db4bbec815b59a959f2
I20240501 20:09:44.988519 25441 ts_tablet_manager.cc:1909] T 46b86f450dbe4db4bbec815b59a959f2 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:09:44.989521 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 46b86f450dbe4db4bbec815b59a959f2 (table rebalancer_dynamic_table_new_name [id=6683ee4f249143d1adbebbee7d1db0e6]) successfully deleted
I20240501 20:09:44.992818 25431 ts_tablet_manager.cc:1398] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20240501 20:09:44.993757 25426 tablet_bootstrap.cc:492] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:44.993937 25426 ts_tablet_manager.cc:1367] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.013s
I20240501 20:09:44.994618 25426 raft_consensus.cc:348] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:44.994854 25426 raft_consensus.cc:374] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.994968 25426 raft_consensus.cc:729] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.995247 25431 tablet_bootstrap.cc:492] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:44.995194 25426 consensus_queue.cc:260] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:44.995759 25426 ts_tablet_manager.cc:1398] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:44.996184 25426 tablet_bootstrap.cc:492] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:44.997769 25426 tablet_bootstrap.cc:654] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.011860 25440 ts_tablet_manager.cc:1888] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:45.012745 25442 ts_tablet_manager.cc:1888] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:45.013437 25442 log.cc:1196] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/46b86f450dbe4db4bbec815b59a959f2
I20240501 20:09:45.014119 25442 ts_tablet_manager.cc:1909] T 46b86f450dbe4db4bbec815b59a959f2 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:45.014711 25440 log.cc:1196] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/46b86f450dbe4db4bbec815b59a959f2
I20240501 20:09:45.016069 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 46b86f450dbe4db4bbec815b59a959f2 (table rebalancer_dynamic_table_new_name [id=6683ee4f249143d1adbebbee7d1db0e6]) successfully deleted
I20240501 20:09:45.016759 25440 ts_tablet_manager.cc:1909] T 46b86f450dbe4db4bbec815b59a959f2 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:45.018338 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 46b86f450dbe4db4bbec815b59a959f2 (table rebalancer_dynamic_table_new_name [id=6683ee4f249143d1adbebbee7d1db0e6]) successfully deleted
I20240501 20:09:45.021664 25426 tablet_bootstrap.cc:492] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:45.021929 25426 ts_tablet_manager.cc:1367] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.013s
I20240501 20:09:45.022595 25426 raft_consensus.cc:348] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:45.022804 25426 raft_consensus.cc:374] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.022889 25426 raft_consensus.cc:729] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.023169 25426 consensus_queue.cc:260] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:45.023728 25426 ts_tablet_manager.cc:1398] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:45.025517 25410 raft_consensus.cc:482] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:45.025732 25410 raft_consensus.cc:504] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:45.026602 25410 leader_election.cc:288] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:45.027272 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1ac1f2f3d642e9a9c785bb4963af1f" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:45.027298 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1ac1f2f3d642e9a9c785bb4963af1f" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:45.027487 23977 raft_consensus.cc:2454] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:45.027601 23710 raft_consensus.cc:2454] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:45.028319 25424 raft_consensus.cc:482] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:45.028499 25424 raft_consensus.cc:504] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.028761 23779 leader_election.cc:302] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:45.029372 25424 leader_election.cc:288] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:45.033362 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25dee912ab5f4613b66d056a8731f188" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:45.033380 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25dee912ab5f4613b66d056a8731f188" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:45.033663 23844 raft_consensus.cc:2454] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:45.033658 23977 raft_consensus.cc:2454] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:45.034273 23644 leader_election.cc:302] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:45.035923 25424 raft_consensus.cc:2781] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:45.036078 25424 raft_consensus.cc:482] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:45.036178 25424 raft_consensus.cc:3037] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.036355 25410 raft_consensus.cc:2781] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:45.036491 25410 raft_consensus.cc:482] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:45.036590 25410 raft_consensus.cc:3037] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.040347 25424 raft_consensus.cc:504] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.041790 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25dee912ab5f4613b66d056a8731f188" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:45.042040 23977 raft_consensus.cc:3037] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.042021 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25dee912ab5f4613b66d056a8731f188" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:45.042294 23844 raft_consensus.cc:3037] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.042634 25424 leader_election.cc:288] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:45.045848 23977 raft_consensus.cc:2454] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:45.046489 23644 leader_election.cc:302] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:45.047022 25424 raft_consensus.cc:2781] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:45.047821 25424 raft_consensus.cc:686] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:45.048313 25424 consensus_queue.cc:237] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.053408 23844 raft_consensus.cc:2454] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:45.053890 25410 raft_consensus.cc:504] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:45.055447 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1ac1f2f3d642e9a9c785bb4963af1f" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:45.055799 23711 raft_consensus.cc:3037] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.059302 23711 raft_consensus.cc:2454] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:45.059965 23781 leader_election.cc:302] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:45.073707 25410 leader_election.cc:288] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:45.074105 25410 raft_consensus.cc:2781] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:45.074337 25410 raft_consensus.cc:686] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:45.078184 25410 consensus_queue.cc:237] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:45.081009 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1ac1f2f3d642e9a9c785bb4963af1f" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:45.081423 23977 raft_consensus.cc:3037] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.103585 25431 tablet_bootstrap.cc:654] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.103940 23977 raft_consensus.cc:2454] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
W20240501 20:09:45.120065 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 60117.3us
I20240501 20:09:45.123152 20818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:45.125202 25431 tablet_bootstrap.cc:492] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:45.125656 25431 ts_tablet_manager.cc:1367] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.131s user 0.003s sys 0.002s
I20240501 20:09:45.128046 20819 catalog_manager.cc:5484] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:45.128837 25431 raft_consensus.cc:348] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:45.129035 25431 raft_consensus.cc:374] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.129127 25431 raft_consensus.cc:729] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.129374 25431 consensus_queue.cc:260] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:45.131867 25431 ts_tablet_manager.cc:1398] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:45.139000 23689 tablet_service.cc:1467] Processing CreateTablet for tablet ac388fd0810549a9835685521828db84 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a2ae9eba47eb4a8aaae22f72660a4608]), partition=
I20240501 20:09:45.139374 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac388fd0810549a9835685521828db84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.144106 20945 tablet_service.cc:1467] Processing CreateTablet for tablet ac388fd0810549a9835685521828db84 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a2ae9eba47eb4a8aaae22f72660a4608]), partition=
I20240501 20:09:45.144469 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac388fd0810549a9835685521828db84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.144551 25138 raft_consensus.cc:482] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:45.144754 25138 raft_consensus.cc:504] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:45.145673 25138 leader_election.cc:288] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:45.146696 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbdc6727cab4983984d49882868014e" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:45.148636 24089 tablet_service.cc:1467] Processing CreateTablet for tablet ac388fd0810549a9835685521828db84 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a2ae9eba47eb4a8aaae22f72660a4608]), partition=
I20240501 20:09:45.148972 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac388fd0810549a9835685521828db84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.152088 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbdc6727cab4983984d49882868014e" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:45.152370 21358 raft_consensus.cc:2379] T fcbdc6727cab4983984d49882868014e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 744c5a532e724391830c0785d32eb2fb in current term 1: Already voted for candidate cda19d6a81c0473db302535ce2ada7f0 in this term.
I20240501 20:09:45.152880 23910 leader_election.cc:302] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb; no voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0
I20240501 20:09:45.153316 25138 raft_consensus.cc:3037] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.157172 25138 raft_consensus.cc:2726] T fcbdc6727cab4983984d49882868014e P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:45.164848 25462 tablet_bootstrap.cc:492] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:45.166615 25462 tablet_bootstrap.cc:654] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.170441 25430 tablet_bootstrap.cc:492] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:45.171291 25427 tablet_bootstrap.cc:492] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:45.172550 25430 tablet_bootstrap.cc:654] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.172766 25427 tablet_bootstrap.cc:654] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
W20240501 20:09:45.173122 23781 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.13.251.190:44663, user_credentials={real_user=slave}} blocked reactor thread for 45565.4us
I20240501 20:09:45.175967 20819 catalog_manager.cc:5484] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:45.177114 25462 tablet_bootstrap.cc:492] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:45.177634 25462 ts_tablet_manager.cc:1367] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 20:09:45.178717 25462 raft_consensus.cc:348] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.180224 25462 raft_consensus.cc:374] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.180524 25462 raft_consensus.cc:729] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.181094 25462 consensus_queue.cc:260] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.183351 25462 ts_tablet_manager.cc:1398] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:45.185672 25427 tablet_bootstrap.cc:492] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:45.185859 25427 ts_tablet_manager.cc:1367] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.001s
I20240501 20:09:45.186703 25427 raft_consensus.cc:348] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.186936 25427 raft_consensus.cc:374] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.187014 25427 raft_consensus.cc:729] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.187363 25427 consensus_queue.cc:260] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.188326 25427 ts_tablet_manager.cc:1398] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:09:45.193688 25430 tablet_bootstrap.cc:492] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:45.193900 25430 ts_tablet_manager.cc:1367] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.006s
I20240501 20:09:45.196254 25430 raft_consensus.cc:348] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.196439 25430 raft_consensus.cc:374] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.197844 25430 raft_consensus.cc:729] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.198179 25430 consensus_queue.cc:260] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.198776 25430 ts_tablet_manager.cc:1398] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 20:09:45.202351 20819 catalog_manager.cc:5484] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:45.214890 25424 raft_consensus.cc:482] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:45.215200 25424 raft_consensus.cc:504] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.216110 25424 leader_election.cc:288] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:45.216264 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac388fd0810549a9835685521828db84" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:45.216578 24108 raft_consensus.cc:2454] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:45.217057 23646 leader_election.cc:302] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:45.217126 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac388fd0810549a9835685521828db84" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:45.217304 25424 raft_consensus.cc:2781] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:45.217393 25424 raft_consensus.cc:482] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:45.217376 20967 raft_consensus.cc:2454] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:45.217500 25424 raft_consensus.cc:3037] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.221014 25424 raft_consensus.cc:504] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.221897 25424 leader_election.cc:288] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:45.222184 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac388fd0810549a9835685521828db84" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:45.222362 24108 raft_consensus.cc:3037] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.222360 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac388fd0810549a9835685521828db84" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:45.222618 20967 raft_consensus.cc:3037] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.225303 24108 raft_consensus.cc:2454] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:45.226034 23646 leader_election.cc:302] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:45.226446 25424 raft_consensus.cc:2781] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:45.226543 20967 raft_consensus.cc:2454] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:45.226639 25424 raft_consensus.cc:686] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:45.226953 25424 consensus_queue.cc:237] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.245565 20819 catalog_manager.cc:5484] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:45.253190 21225 consensus_queue.cc:237] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:45.255304 21358 raft_consensus.cc:1261] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:45.255307 24108 raft_consensus.cc:1261] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:45.255833 25432 consensus_queue.cc:1035] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:45.256170 25429 consensus_queue.cc:1035] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:45.259018 25074 raft_consensus.cc:2932] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:45.259665 24108 raft_consensus.cc:2932] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:45.266783 20819 catalog_manager.cc:5484] T c1e295a46a174f3f8673e03f57890b58 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:45.272234 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e295a46a174f3f8673e03f57890b58 with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:45.272588 25442 tablet_replica.cc:330] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:45.272799 25442 raft_consensus.cc:2229] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:45.272984 25442 raft_consensus.cc:2258] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:45.273504 25442 ts_tablet_manager.cc:1875] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:45.279210 25442 ts_tablet_manager.cc:1888] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:09:45.279423 25442 log.cc:1196] T c1e295a46a174f3f8673e03f57890b58 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/c1e295a46a174f3f8673e03f57890b58
I20240501 20:09:45.280429 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet c1e295a46a174f3f8673e03f57890b58 (table rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]) successfully deleted
I20240501 20:09:45.280387 21357 raft_consensus.cc:2932] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:45.361950 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:45.370836 20967 raft_consensus.cc:1261] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:45.371989 24108 raft_consensus.cc:1261] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:45.371373 25469 consensus_queue.cc:1035] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.374703 25469 consensus_queue.cc:1035] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.391111 20819 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet ac388fd0810549a9835685521828db84 (table rebalancer_dynamic_table [id=a2ae9eba47eb4a8aaae22f72660a4608]). Expected version 1 got 0
I20240501 20:09:45.395210 25508 tablet.cc:1722] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:45.395378 20822 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet ac388fd0810549a9835685521828db84 (table rebalancer_dynamic_table [id=a2ae9eba47eb4a8aaae22f72660a4608]). Expected version 1 got 0
I20240501 20:09:45.396201 25508 tablet.cc:1620] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:45.396862 25510 tablet.cc:1722] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:45.397374 25510 tablet.cc:1620] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:45.407589 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=a2ae9eba47eb4a8aaae22f72660a4608] alter complete (version 1)
I20240501 20:09:45.408390 20823 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet ac388fd0810549a9835685521828db84 (table rebalancer_dynamic_table [id=a2ae9eba47eb4a8aaae22f72660a4608]). Expected version 1 got 0
I20240501 20:09:45.410197 25148 consensus_queue.cc:1035] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.418727 25148 consensus_queue.cc:1035] T fcbdc6727cab4983984d49882868014e P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.438702 25515 tablet.cc:1722] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:45.439764 25515 tablet.cc:1620] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:45.440891 25469 consensus_queue.cc:1035] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.457149 20823 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet ac388fd0810549a9835685521828db84 (table rebalancer_dynamic_table [id=a2ae9eba47eb4a8aaae22f72660a4608]). Expected version 1 got 0
I20240501 20:09:45.482141 25469 consensus_queue.cc:1035] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.512516 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:45.546278 25508 tablet.cc:1722] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:45.546986 25508 tablet.cc:1620] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:45.553043 25515 tablet.cc:1722] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:45.553776 25515 tablet.cc:1620] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:45.559334 25510 tablet.cc:1722] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:45.559913 25510 tablet.cc:1620] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
W20240501 20:09:45.598014 24046 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 65560.5us
I20240501 20:09:45.608958 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=a2ae9eba47eb4a8aaae22f72660a4608] alter complete (version 2)
I20240501 20:09:45.626441 25410 consensus_queue.cc:1035] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.642992 25460 consensus_queue.cc:1035] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.676605 21208 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:45.714030 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:45.725535 25508 tablet.cc:1722] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:45.726329 25515 tablet.cc:1722] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:45.731669 25510 tablet.cc:1722] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:45.742570 23956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:45.745858 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=a2ae9eba47eb4a8aaae22f72660a4608] alter complete (version 3)
I20240501 20:09:45.764914 23689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:45.769810 21336 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:09:45.786489 20822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:45.800017 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1a90f2878c433381aed777ac33551d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.800155 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1a90f2878c433381aed777ac33551d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.800536 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1a90f2878c433381aed777ac33551d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.800659 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1a90f2878c433381aed777ac33551d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.803058 24089 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:45.807735 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 182bd3935cce49868a451685097e9f0c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.808190 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182bd3935cce49868a451685097e9f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.810026 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1a90f2878c433381aed777ac33551d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.810423 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1a90f2878c433381aed777ac33551d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.812953 23825 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:45.819444 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 2b57a80759944301a033140e482aec36 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.819957 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b57a80759944301a033140e482aec36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.820272 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 182bd3935cce49868a451685097e9f0c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.820722 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182bd3935cce49868a451685097e9f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.821875 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 2b57a80759944301a033140e482aec36 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.822219 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b57a80759944301a033140e482aec36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.823952 23690 tablet_service.cc:1467] Processing CreateTablet for tablet 2b57a80759944301a033140e482aec36 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.824384 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b57a80759944301a033140e482aec36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.856302 20818 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:45.856565 20818 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:45.860980 20818 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet ac388fd0810549a9835685521828db84
I20240501 20:09:45.863104 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet ac388fd0810549a9835685521828db84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:45 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:45.865356 23690 tablet_service.cc:1514] Processing DeleteTablet for tablet ac388fd0810549a9835685521828db84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:45 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:45.865938 25553 tablet_bootstrap.cc:492] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:45.868441 25553 tablet_bootstrap.cc:654] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.877169 25554 tablet_bootstrap.cc:492] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:45.882119 25554 tablet_bootstrap.cc:654] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.883489 25555 tablet_bootstrap.cc:492] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:45.889218 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 182bd3935cce49868a451685097e9f0c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.889737 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182bd3935cce49868a451685097e9f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.892130 21076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:45.899104 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet ac388fd0810549a9835685521828db84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:45 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:45.909389 25559 tablet_replica.cc:330] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:45.909941 25559 raft_consensus.cc:2229] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:45.910509 25559 raft_consensus.cc:2258] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:45.901683 25558 tablet_replica.cc:330] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:45.911907 25558 raft_consensus.cc:2229] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:45.912288 25558 raft_consensus.cc:2258] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:45.913452 25558 ts_tablet_manager.cc:1875] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:45.921624 25555 tablet_bootstrap.cc:654] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.923519 25562 tablet_bootstrap.cc:492] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:45.925769 25562 tablet_bootstrap.cc:654] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.928499 25556 tablet_bootstrap.cc:492] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:45.935429 25556 tablet_bootstrap.cc:654] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.938549 25555 tablet_bootstrap.cc:492] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:45.940002 25554 tablet_bootstrap.cc:492] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:45.940253 25554 ts_tablet_manager.cc:1367] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.003s
I20240501 20:09:45.941179 25554 raft_consensus.cc:348] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.941550 25554 raft_consensus.cc:374] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.941684 25554 raft_consensus.cc:729] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.942013 25554 consensus_queue.cc:260] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.942909 25554 ts_tablet_manager.cc:1398] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:45.943173 25555 ts_tablet_manager.cc:1367] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.003s
I20240501 20:09:45.944579 25562 tablet_bootstrap.cc:492] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:45.945979 25553 tablet_bootstrap.cc:492] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:45.946149 25553 ts_tablet_manager.cc:1367] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.000s
I20240501 20:09:45.947117 25553 raft_consensus.cc:348] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.947468 25553 raft_consensus.cc:374] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.947644 25553 raft_consensus.cc:729] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.948011 25553 consensus_queue.cc:260] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.949925 25553 ts_tablet_manager.cc:1398] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:09:45.950350 25553 tablet_bootstrap.cc:492] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:45.951260 25559 ts_tablet_manager.cc:1875] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:45.958613 25553 tablet_bootstrap.cc:654] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.960299 25555 raft_consensus.cc:348] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.960541 25555 raft_consensus.cc:374] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.960652 25555 raft_consensus.cc:729] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.960916 25555 consensus_queue.cc:260] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.961719 25555 ts_tablet_manager.cc:1398] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:45.962896 25555 tablet_bootstrap.cc:492] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:45.964485 25563 tablet_replica.cc:330] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:45.964576 25562 ts_tablet_manager.cc:1367] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20240501 20:09:45.964721 25563 raft_consensus.cc:2229] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:45.965395 20818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:45.965999 25563 raft_consensus.cc:2258] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:45.967257 25563 ts_tablet_manager.cc:1875] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:45.970130 25558 ts_tablet_manager.cc:1888] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:45.965550 25562 raft_consensus.cc:348] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.970636 25562 raft_consensus.cc:374] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.970824 25562 raft_consensus.cc:729] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.972285 25562 consensus_queue.cc:260] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:45.973695 25562 ts_tablet_manager.cc:1398] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:09:45.974905 25555 tablet_bootstrap.cc:654] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.975260 25553 tablet_bootstrap.cc:492] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:45.975453 25559 ts_tablet_manager.cc:1888] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:45.975502 25553 ts_tablet_manager.cc:1367] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.006s
I20240501 20:09:45.976677 25559 log.cc:1196] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/ac388fd0810549a9835685521828db84
I20240501 20:09:45.977262 25559 ts_tablet_manager.cc:1909] T ac388fd0810549a9835685521828db84 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:45.977481 25558 log.cc:1196] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/ac388fd0810549a9835685521828db84
I20240501 20:09:45.978929 25558 ts_tablet_manager.cc:1909] T ac388fd0810549a9835685521828db84 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:45.976562 25556 tablet_bootstrap.cc:492] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:45.979211 25556 ts_tablet_manager.cc:1367] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20240501 20:09:45.980121 25556 raft_consensus.cc:348] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.980487 25556 raft_consensus.cc:374] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.980640 25556 raft_consensus.cc:729] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.980919 25556 consensus_queue.cc:260] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.983225 25556 ts_tablet_manager.cc:1398] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:45.983788 25556 tablet_bootstrap.cc:492] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:45.985520 25556 tablet_bootstrap.cc:654] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.986804 24088 tablet_service.cc:1467] Processing CreateTablet for tablet ba68e221df194e54903a434b7bfca3bf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8015b82ccd0d473cb5d1c9506b3ff27b]), partition=
I20240501 20:09:45.987097 23957 tablet_service.cc:1467] Processing CreateTablet for tablet ba68e221df194e54903a434b7bfca3bf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8015b82ccd0d473cb5d1c9506b3ff27b]), partition=
I20240501 20:09:45.987224 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba68e221df194e54903a434b7bfca3bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.987444 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba68e221df194e54903a434b7bfca3bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.990906 25553 raft_consensus.cc:348] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.991168 25553 raft_consensus.cc:374] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.991374 25553 raft_consensus.cc:729] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.991880 25553 consensus_queue.cc:260] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.992184 25138 raft_consensus.cc:482] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:45.992341 25138 raft_consensus.cc:504] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:45.993022 23824 tablet_service.cc:1467] Processing CreateTablet for tablet ba68e221df194e54903a434b7bfca3bf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8015b82ccd0d473cb5d1c9506b3ff27b]), partition=
I20240501 20:09:45.993095 25138 leader_election.cc:288] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:45.993357 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba68e221df194e54903a434b7bfca3bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.994284 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1a90f2878c433381aed777ac33551d" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:45.994556 23846 raft_consensus.cc:2454] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:45.994913 25554 tablet_bootstrap.cc:492] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:45.995031 23912 leader_election.cc:302] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:45.995577 25138 raft_consensus.cc:2781] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:45.995700 25138 raft_consensus.cc:482] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:45.995848 25138 raft_consensus.cc:3037] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.996749 25554 tablet_bootstrap.cc:654] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.000263 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet ac388fd0810549a9835685521828db84 (table rebalancer_dynamic_table_new_name [id=a2ae9eba47eb4a8aaae22f72660a4608]) successfully deleted
I20240501 20:09:46.000567 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet ac388fd0810549a9835685521828db84 (table rebalancer_dynamic_table_new_name [id=a2ae9eba47eb4a8aaae22f72660a4608]) successfully deleted
I20240501 20:09:45.995738 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1a90f2878c433381aed777ac33551d" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:46.001420 23711 raft_consensus.cc:2454] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:46.002993 25138 raft_consensus.cc:504] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.005718 25138 leader_election.cc:288] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:46.008725 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1a90f2878c433381aed777ac33551d" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:46.008984 23711 raft_consensus.cc:3037] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.009763 25553 ts_tablet_manager.cc:1398] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.034s user 0.000s sys 0.002s
I20240501 20:09:46.012099 23711 raft_consensus.cc:2454] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:46.012584 23911 leader_election.cc:302] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:46.012859 25138 raft_consensus.cc:2781] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:46.013017 25138 raft_consensus.cc:686] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:46.013314 25138 consensus_queue.cc:237] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.017604 25563 ts_tablet_manager.cc:1888] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:46.017815 25563 log.cc:1196] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/ac388fd0810549a9835685521828db84
I20240501 20:09:46.018218 25553 tablet_bootstrap.cc:492] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:46.018368 25563 ts_tablet_manager.cc:1909] T ac388fd0810549a9835685521828db84 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:46.019608 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet ac388fd0810549a9835685521828db84 (table rebalancer_dynamic_table_new_name [id=a2ae9eba47eb4a8aaae22f72660a4608]) successfully deleted
I20240501 20:09:46.024616 20945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:46.024784 25532 raft_consensus.cc:482] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:46.024973 25532 raft_consensus.cc:504] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.025871 25532 leader_election.cc:288] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:46.026674 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b57a80759944301a033140e482aec36" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:09:46.027316 24046 leader_election.cc:341] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:46.027380 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b57a80759944301a033140e482aec36" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
W20240501 20:09:46.027935 24045 leader_election.cc:341] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:46.028079 24045 leader_election.cc:302] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:46.029531 25532 raft_consensus.cc:2726] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:46.031359 25554 tablet_bootstrap.cc:492] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:46.031679 25554 ts_tablet_manager.cc:1367] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.003s
I20240501 20:09:46.032891 25554 raft_consensus.cc:348] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.034441 25554 raft_consensus.cc:374] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.034627 25554 raft_consensus.cc:729] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.034879 25556 tablet_bootstrap.cc:492] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:46.035604 25556 ts_tablet_manager.cc:1367] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240501 20:09:46.035318 25554 consensus_queue.cc:260] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.038827 25554 ts_tablet_manager.cc:1398] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:09:46.039875 25556 raft_consensus.cc:348] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.041838 25556 raft_consensus.cc:374] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.041978 25556 raft_consensus.cc:729] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.042274 25556 consensus_queue.cc:260] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.043183 25556 ts_tablet_manager.cc:1398] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:09:46.044250 25555 tablet_bootstrap.cc:492] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:46.044476 25555 ts_tablet_manager.cc:1367] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.082s user 0.001s sys 0.005s
I20240501 20:09:46.044656 25553 tablet_bootstrap.cc:654] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.045327 25555 raft_consensus.cc:348] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.045537 25555 raft_consensus.cc:374] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.045624 25555 raft_consensus.cc:729] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.045892 25555 consensus_queue.cc:260] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.046608 25555 ts_tablet_manager.cc:1398] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:46.046940 25555 tablet_bootstrap.cc:492] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:46.047518 25556 tablet_bootstrap.cc:492] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:46.050530 25555 tablet_bootstrap.cc:654] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.051648 25556 tablet_bootstrap.cc:654] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.067416 25556 tablet_bootstrap.cc:492] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:46.067459 25553 tablet_bootstrap.cc:492] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:46.067730 25556 ts_tablet_manager.cc:1367] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20240501 20:09:46.068771 25556 raft_consensus.cc:348] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.069079 25556 raft_consensus.cc:374] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.069248 25556 raft_consensus.cc:729] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.069602 25556 consensus_queue.cc:260] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.071363 20819 catalog_manager.cc:5484] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:46.072547 25553 ts_tablet_manager.cc:1367] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.002s
I20240501 20:09:46.073379 25553 raft_consensus.cc:348] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.075137 25553 raft_consensus.cc:374] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.075304 25553 raft_consensus.cc:729] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.075723 25553 consensus_queue.cc:260] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.077502 25553 ts_tablet_manager.cc:1398] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:46.083096 25556 ts_tablet_manager.cc:1398] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240501 20:09:46.095685 25555 tablet_bootstrap.cc:492] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:46.095988 25555 ts_tablet_manager.cc:1367] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.012s
I20240501 20:09:46.096899 25555 raft_consensus.cc:348] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.097184 25555 raft_consensus.cc:374] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.097298 25555 raft_consensus.cc:729] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.098011 25555 consensus_queue.cc:260] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.099802 25555 ts_tablet_manager.cc:1398] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240501 20:09:46.112221 23912 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 117183us
I20240501 20:09:46.113476 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1a90f2878c433381aed777ac33551d" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:46.113698 23846 raft_consensus.cc:3037] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.119105 23846 raft_consensus.cc:2454] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:46.157208 25570 raft_consensus.cc:482] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:46.157397 25570 raft_consensus.cc:504] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.158219 25570 leader_election.cc:288] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:46.158670 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba68e221df194e54903a434b7bfca3bf" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:46.158888 24108 raft_consensus.cc:2454] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:46.159262 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba68e221df194e54903a434b7bfca3bf" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:46.159327 23911 leader_election.cc:302] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:46.159509 23846 raft_consensus.cc:2454] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:46.159628 25570 raft_consensus.cc:2781] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:46.159725 25570 raft_consensus.cc:482] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:46.159809 25570 raft_consensus.cc:3037] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.161974 25570 raft_consensus.cc:504] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.162972 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba68e221df194e54903a434b7bfca3bf" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:46.163173 24108 raft_consensus.cc:3037] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.163245 25570 leader_election.cc:288] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:46.163717 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba68e221df194e54903a434b7bfca3bf" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:46.163934 23846 raft_consensus.cc:3037] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.166635 24108 raft_consensus.cc:2454] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:46.166651 23846 raft_consensus.cc:2454] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:46.167064 23911 leader_election.cc:302] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:46.167419 25570 raft_consensus.cc:2781] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:46.167568 25570 raft_consensus.cc:686] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:46.167861 25570 consensus_queue.cc:237] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.171149 20819 catalog_manager.cc:5484] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:46.195978 25410 raft_consensus.cc:482] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:46.196192 25410 raft_consensus.cc:504] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.196985 25410 leader_election.cc:288] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:46.197613 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b57a80759944301a033140e482aec36" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:46.197820 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b57a80759944301a033140e482aec36" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:46.197923 24108 raft_consensus.cc:2454] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:46.198074 23711 raft_consensus.cc:2454] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:46.198421 23781 leader_election.cc:302] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:46.199097 25410 raft_consensus.cc:2781] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:46.199250 25410 raft_consensus.cc:482] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:46.199354 25410 raft_consensus.cc:3037] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.202795 25410 raft_consensus.cc:504] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.204586 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b57a80759944301a033140e482aec36" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:46.204799 23711 raft_consensus.cc:3037] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.210019 23711 raft_consensus.cc:2454] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:46.212746 23781 leader_election.cc:302] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:46.213074 25460 raft_consensus.cc:2781] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:46.218901 25460 raft_consensus.cc:686] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:46.219308 25460 consensus_queue.cc:237] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:46.225875 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b57a80759944301a033140e482aec36" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:46.226197 24108 raft_consensus.cc:3037] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.227005 25410 leader_election.cc:288] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:46.227473 20819 catalog_manager.cc:5484] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:46.229914 24108 raft_consensus.cc:2454] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:46.230541 23977 consensus_queue.cc:237] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.233318 24108 raft_consensus.cc:1261] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:46.233325 20967 raft_consensus.cc:1261] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:46.233892 25138 consensus_queue.cc:1035] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:46.234468 25138 consensus_queue.cc:1035] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:46.236219 25570 raft_consensus.cc:482] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:46.236428 25570 raft_consensus.cc:504] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.237092 25570 leader_election.cc:288] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:46.237526 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182bd3935cce49868a451685097e9f0c" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:46.237794 23711 raft_consensus.cc:2454] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:46.237851 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182bd3935cce49868a451685097e9f0c" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:46.238065 20964 raft_consensus.cc:2454] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:46.238131 25570 raft_consensus.cc:2932] T 2ded048b4bb94eb79f95554db7e15b7b P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:46.238794 23909 leader_election.cc:302] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:46.239203 25574 raft_consensus.cc:2781] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:46.239310 25574 raft_consensus.cc:482] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:46.239446 25574 raft_consensus.cc:3037] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.242607 25574 raft_consensus.cc:504] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.243266 24108 raft_consensus.cc:2932] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:46.243830 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182bd3935cce49868a451685097e9f0c" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:46.244062 20964 raft_consensus.cc:3037] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.243851 20967 raft_consensus.cc:2932] T 2ded048b4bb94eb79f95554db7e15b7b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:46.246912 20964 raft_consensus.cc:2454] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:46.251039 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182bd3935cce49868a451685097e9f0c" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:46.251305 23711 raft_consensus.cc:3037] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.253854 20819 catalog_manager.cc:5484] T 2ded048b4bb94eb79f95554db7e15b7b P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:46.254069 23711 raft_consensus.cc:2454] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:46.255398 23909 leader_election.cc:302] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:46.255856 25570 raft_consensus.cc:2781] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:46.256314 25570 raft_consensus.cc:686] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:46.256628 25574 leader_election.cc:288] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:46.256687 25570 consensus_queue.cc:237] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.260221 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ded048b4bb94eb79f95554db7e15b7b with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:46.262861 25606 tablet_replica.cc:330] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:46.263101 25606 raft_consensus.cc:2229] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:46.263386 25606 raft_consensus.cc:2258] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:46.264112 25606 ts_tablet_manager.cc:1875] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:46.263891 20818 catalog_manager.cc:5484] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:46.270668 25606 ts_tablet_manager.cc:1888] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:09:46.270839 25606 log.cc:1196] T 2ded048b4bb94eb79f95554db7e15b7b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/2ded048b4bb94eb79f95554db7e15b7b
I20240501 20:09:46.271723 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 2ded048b4bb94eb79f95554db7e15b7b (table rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]) successfully deleted
I20240501 20:09:46.281656 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:46.288688 23846 raft_consensus.cc:1261] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:46.289526 25574 consensus_queue.cc:1035] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.290494 24108 raft_consensus.cc:1261] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:46.291937 25574 consensus_queue.cc:1035] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.298358 20818 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet ba68e221df194e54903a434b7bfca3bf (table rebalancer_dynamic_table [id=8015b82ccd0d473cb5d1c9506b3ff27b]). Expected version 1 got 0
I20240501 20:09:46.316746 25620 tablet.cc:1722] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:46.317080 20818 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet ba68e221df194e54903a434b7bfca3bf (table rebalancer_dynamic_table [id=8015b82ccd0d473cb5d1c9506b3ff27b]). Expected version 1 got 0
I20240501 20:09:46.317514 25620 tablet.cc:1620] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:46.318248 25621 tablet.cc:1722] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:46.318754 25621 tablet.cc:1620] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:46.320744 20818 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet ba68e221df194e54903a434b7bfca3bf (table rebalancer_dynamic_table [id=8015b82ccd0d473cb5d1c9506b3ff27b]). Expected version 1 got 0
I20240501 20:09:46.324178 25623 tablet.cc:1722] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:46.324661 25623 tablet.cc:1620] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:46.327806 20819 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=8015b82ccd0d473cb5d1c9506b3ff27b] alter complete (version 1)
I20240501 20:09:46.336580 20818 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet ba68e221df194e54903a434b7bfca3bf (table rebalancer_dynamic_table [id=8015b82ccd0d473cb5d1c9506b3ff27b]). Expected version 1 got 0
I20240501 20:09:46.432291 20825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:46.448684 25620 tablet.cc:1722] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:46.449188 25620 tablet.cc:1620] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:46.452481 25623 tablet.cc:1722] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:46.452903 25623 tablet.cc:1620] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:46.462721 25621 tablet.cc:1722] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:46.463342 25621 tablet.cc:1620] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:46.463572 25138 consensus_queue.cc:1035] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.467738 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=8015b82ccd0d473cb5d1c9506b3ff27b] alter complete (version 2)
I20240501 20:09:46.471683 25574 consensus_queue.cc:1035] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.541173 21208 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:46.576052 23689 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:46.581076 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:46.593361 25620 tablet.cc:1722] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:46.594591 25621 tablet.cc:1722] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:46.596187 25623 tablet.cc:1722] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:46.605854 20825 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=8015b82ccd0d473cb5d1c9506b3ff27b] alter complete (version 3)
I20240501 20:09:46.657089 21336 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:09:46.662786 23957 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:46.675645 25460 consensus_queue.cc:1035] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.689548 25410 consensus_queue.cc:1035] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.698982 25138 consensus_queue.cc:1035] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.710681 20825 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:46.711007 20825 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:46.718349 20825 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet ba68e221df194e54903a434b7bfca3bf
I20240501 20:09:46.719790 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet ba68e221df194e54903a434b7bfca3bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:46 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:46.720218 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet ba68e221df194e54903a434b7bfca3bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:46 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:46.723814 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet ba68e221df194e54903a434b7bfca3bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:46 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:46.724109 25606 tablet_replica.cc:330] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:46.724298 25606 raft_consensus.cc:2229] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:46.724509 25606 raft_consensus.cc:2258] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:46.725422 25606 ts_tablet_manager.cc:1875] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:46.731746 25675 tablet_replica.cc:330] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:46.731998 25675 raft_consensus.cc:2229] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:46.732419 25675 raft_consensus.cc:2258] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:46.732980 25674 tablet_replica.cc:330] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:46.733201 25674 raft_consensus.cc:2229] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:46.733441 25674 raft_consensus.cc:2258] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:46.741122 23824 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:46.751586 25674 ts_tablet_manager.cc:1875] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:46.752703 25675 ts_tablet_manager.cc:1875] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:46.762732 25606 ts_tablet_manager.cc:1888] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:46.762920 25606 log.cc:1196] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/ba68e221df194e54903a434b7bfca3bf
I20240501 20:09:46.763509 25606 ts_tablet_manager.cc:1909] T ba68e221df194e54903a434b7bfca3bf P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:09:46.764916 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet ba68e221df194e54903a434b7bfca3bf (table rebalancer_dynamic_table_new_name [id=8015b82ccd0d473cb5d1c9506b3ff27b]) successfully deleted
I20240501 20:09:46.765419 25570 consensus_queue.cc:1035] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.770068 20947 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:46.777256 24088 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:46.783071 25674 ts_tablet_manager.cc:1888] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:46.788169 25674 log.cc:1196] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/ba68e221df194e54903a434b7bfca3bf
I20240501 20:09:46.788880 25674 ts_tablet_manager.cc:1909] T ba68e221df194e54903a434b7bfca3bf P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:46.790249 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet ba68e221df194e54903a434b7bfca3bf (table rebalancer_dynamic_table_new_name [id=8015b82ccd0d473cb5d1c9506b3ff27b]) successfully deleted
I20240501 20:09:46.791254 25675 ts_tablet_manager.cc:1888] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:46.791589 25675 log.cc:1196] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/ba68e221df194e54903a434b7bfca3bf
I20240501 20:09:46.792196 25675 ts_tablet_manager.cc:1909] T ba68e221df194e54903a434b7bfca3bf P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:46.793351 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet ba68e221df194e54903a434b7bfca3bf (table rebalancer_dynamic_table_new_name [id=8015b82ccd0d473cb5d1c9506b3ff27b]) successfully deleted
I20240501 20:09:46.795239 21076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:46.814123 20818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:46.830376 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 83384d743f874d77bc8a12d1c9834736 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.830862 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83384d743f874d77bc8a12d1c9834736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.831408 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 83384d743f874d77bc8a12d1c9834736 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.831794 20818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:46.832139 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83384d743f874d77bc8a12d1c9834736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.834131 23690 tablet_service.cc:1467] Processing CreateTablet for tablet 74f23f1dafeb4447b33cb7f5bdbf03c4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.834507 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f23f1dafeb4447b33cb7f5bdbf03c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.837919 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 83384d743f874d77bc8a12d1c9834736 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.839249 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 74f23f1dafeb4447b33cb7f5bdbf03c4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.839926 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 74f23f1dafeb4447b33cb7f5bdbf03c4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.840358 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f23f1dafeb4447b33cb7f5bdbf03c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.840860 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 718853c0da3d4a68ab478a6e79cdd438 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.841645 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f23f1dafeb4447b33cb7f5bdbf03c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.842185 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83384d743f874d77bc8a12d1c9834736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.842607 20946 tablet_service.cc:1467] Processing CreateTablet for tablet 718853c0da3d4a68ab478a6e79cdd438 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.842968 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718853c0da3d4a68ab478a6e79cdd438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.846007 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718853c0da3d4a68ab478a6e79cdd438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.846304 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 718853c0da3d4a68ab478a6e79cdd438 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.846741 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718853c0da3d4a68ab478a6e79cdd438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.862200 25681 tablet_bootstrap.cc:492] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:46.862712 23825 tablet_service.cc:1467] Processing CreateTablet for tablet eaaf183dac384f179b7abbade02a8d03 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ead7a0225924dc8ac72634a63383e36]), partition=
I20240501 20:09:46.862936 23691 tablet_service.cc:1467] Processing CreateTablet for tablet eaaf183dac384f179b7abbade02a8d03 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ead7a0225924dc8ac72634a63383e36]), partition=
I20240501 20:09:46.863076 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaaf183dac384f179b7abbade02a8d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.863245 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaaf183dac384f179b7abbade02a8d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.868350 24088 tablet_service.cc:1467] Processing CreateTablet for tablet eaaf183dac384f179b7abbade02a8d03 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ead7a0225924dc8ac72634a63383e36]), partition=
I20240501 20:09:46.868713 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaaf183dac384f179b7abbade02a8d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.867525 25681 tablet_bootstrap.cc:654] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.879688 25685 tablet_bootstrap.cc:492] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:46.883450 25685 tablet_bootstrap.cc:654] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.882354 25684 tablet_bootstrap.cc:492] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:46.885186 25683 tablet_bootstrap.cc:492] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:46.889705 25686 tablet_bootstrap.cc:492] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:46.892177 25685 tablet_bootstrap.cc:492] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:46.892196 25680 tablet_bootstrap.cc:492] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:46.892781 25685 ts_tablet_manager.cc:1367] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 20:09:46.893682 25684 tablet_bootstrap.cc:654] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.893791 25686 tablet_bootstrap.cc:654] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.893805 25685 raft_consensus.cc:348] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.894112 25685 raft_consensus.cc:374] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.894279 25685 raft_consensus.cc:729] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.894613 25685 consensus_queue.cc:260] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.895418 25685 ts_tablet_manager.cc:1398] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:46.896179 25685 tablet_bootstrap.cc:492] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:46.897241 25683 tablet_bootstrap.cc:654] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.898890 25685 tablet_bootstrap.cc:654] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.920442 25680 tablet_bootstrap.cc:654] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.924598 25681 tablet_bootstrap.cc:492] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:46.926242 25681 ts_tablet_manager.cc:1367] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.000s
I20240501 20:09:46.927321 25681 raft_consensus.cc:348] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:46.927615 25684 tablet_bootstrap.cc:492] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:46.927726 25681 raft_consensus.cc:374] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.927790 25684 ts_tablet_manager.cc:1367] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.009s
I20240501 20:09:46.927935 25681 raft_consensus.cc:729] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.928319 25681 consensus_queue.cc:260] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:46.928606 25686 tablet_bootstrap.cc:492] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:46.928742 25683 tablet_bootstrap.cc:492] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:46.929319 25683 ts_tablet_manager.cc:1367] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.007s
I20240501 20:09:46.929973 25686 ts_tablet_manager.cc:1367] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20240501 20:09:46.930227 25683 raft_consensus.cc:348] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.930428 25683 raft_consensus.cc:374] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.930543 25683 raft_consensus.cc:729] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.936630 25680 tablet_bootstrap.cc:492] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:46.931532 25686 raft_consensus.cc:348] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.940052 25680 ts_tablet_manager.cc:1367] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.003s
I20240501 20:09:46.940212 25686 raft_consensus.cc:374] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.941165 25680 raft_consensus.cc:348] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.941498 25686 raft_consensus.cc:729] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.941974 25685 tablet_bootstrap.cc:492] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:46.942140 25685 ts_tablet_manager.cc:1367] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.005s
I20240501 20:09:46.942350 25686 consensus_queue.cc:260] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.942831 25685 raft_consensus.cc:348] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.943044 25685 raft_consensus.cc:374] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.943029 25681 ts_tablet_manager.cc:1398] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 20:09:46.943102 25685 raft_consensus.cc:729] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.943305 25685 consensus_queue.cc:260] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.944110 25684 raft_consensus.cc:348] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:46.944325 25684 raft_consensus.cc:374] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.944403 25684 raft_consensus.cc:729] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.944607 25684 consensus_queue.cc:260] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:46.945024 25683 consensus_queue.cc:260] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.946211 25684 ts_tablet_manager.cc:1398] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240501 20:09:46.947713 25685 ts_tablet_manager.cc:1398] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 20:09:46.948029 25685 tablet_bootstrap.cc:492] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:46.945647 25683 ts_tablet_manager.cc:1398] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240501 20:09:46.949026 25683 tablet_bootstrap.cc:492] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:46.950757 25685 tablet_bootstrap.cc:654] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.952045 25680 raft_consensus.cc:374] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.951941 25686 ts_tablet_manager.cc:1398] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 20:09:46.952615 25680 raft_consensus.cc:729] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.952971 25683 tablet_bootstrap.cc:654] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.955129 25686 tablet_bootstrap.cc:492] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:46.955494 25680 consensus_queue.cc:260] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:46.958787 25680 ts_tablet_manager.cc:1398] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240501 20:09:46.960069 25680 tablet_bootstrap.cc:492] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:46.966290 25686 tablet_bootstrap.cc:654] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.968109 25680 tablet_bootstrap.cc:654] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.976200 25686 tablet_bootstrap.cc:492] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:46.976439 25686 ts_tablet_manager.cc:1367] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.001s
I20240501 20:09:46.977245 25686 raft_consensus.cc:348] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:46.977569 25686 raft_consensus.cc:374] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.977686 25686 raft_consensus.cc:729] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.977998 25686 consensus_queue.cc:260] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:46.993294 25686 ts_tablet_manager.cc:1398] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240501 20:09:46.994529 25680 tablet_bootstrap.cc:492] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:46.994764 25680 ts_tablet_manager.cc:1367] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.000s
I20240501 20:09:46.996416 25680 raft_consensus.cc:348] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:46.996754 25680 raft_consensus.cc:374] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.997035 25680 raft_consensus.cc:729] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.997381 25680 consensus_queue.cc:260] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:46.998150 25680 ts_tablet_manager.cc:1398] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:47.018337 25688 raft_consensus.cc:482] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.018595 25688 raft_consensus.cc:504] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:47.021597 25688 leader_election.cc:288] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:47.024412 25424 raft_consensus.cc:482] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.024611 25424 raft_consensus.cc:504] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.025435 25424 leader_election.cc:288] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:47.035077 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83384d743f874d77bc8a12d1c9834736" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:47.035392 23977 raft_consensus.cc:2454] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:47.037598 23644 leader_election.cc:302] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:47.037891 25701 raft_consensus.cc:2781] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:47.038080 25701 raft_consensus.cc:482] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:47.038264 25701 raft_consensus.cc:3037] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.043026 25701 raft_consensus.cc:504] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.049374 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718853c0da3d4a68ab478a6e79cdd438" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:47.049636 23977 raft_consensus.cc:2454] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:47.050025 21161 leader_election.cc:302] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:47.050293 25688 raft_consensus.cc:2781] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:47.050406 25688 raft_consensus.cc:482] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:47.050503 25688 raft_consensus.cc:3037] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.050792 25701 leader_election.cc:288] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:47.051604 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83384d743f874d77bc8a12d1c9834736" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:47.051820 23977 raft_consensus.cc:3037] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.053341 25688 raft_consensus.cc:504] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:47.053947 25688 leader_election.cc:288] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:47.054567 23977 raft_consensus.cc:2454] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:47.054515 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718853c0da3d4a68ab478a6e79cdd438" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:47.054765 23976 raft_consensus.cc:3037] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.055019 23644 leader_election.cc:302] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:47.055297 25701 raft_consensus.cc:2781] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:47.055599 25701 raft_consensus.cc:686] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:47.055953 25701 consensus_queue.cc:237] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.057598 23976 raft_consensus.cc:2454] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:47.059177 21161 leader_election.cc:302] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:47.059860 25688 raft_consensus.cc:2781] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:47.060012 25688 raft_consensus.cc:686] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:47.060309 25688 consensus_queue.cc:237] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:47.062268 20825 catalog_manager.cc:5484] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
W20240501 20:09:47.107661 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 122164us
I20240501 20:09:47.108810 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718853c0da3d4a68ab478a6e79cdd438" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:47.109649 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718853c0da3d4a68ab478a6e79cdd438" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
W20240501 20:09:47.111429 21161 leader_election.cc:341] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:47.112205 21161 leader_election.cc:341] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:47.115798 25685 tablet_bootstrap.cc:492] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:47.115986 25685 ts_tablet_manager.cc:1367] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.168s user 0.022s sys 0.031s
I20240501 20:09:47.117893 25685 raft_consensus.cc:348] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:47.118160 25685 raft_consensus.cc:374] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.118268 25685 raft_consensus.cc:729] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.118566 25685 consensus_queue.cc:260] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:47.119400 25683 tablet_bootstrap.cc:492] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:47.119596 25683 ts_tablet_manager.cc:1367] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.171s user 0.024s sys 0.029s
I20240501 20:09:47.120348 25683 raft_consensus.cc:348] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:47.120574 25683 raft_consensus.cc:374] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.120656 25683 raft_consensus.cc:729] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.120926 25683 consensus_queue.cc:260] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:47.123059 25683 ts_tablet_manager.cc:1398] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:47.123354 25683 tablet_bootstrap.cc:492] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:47.123796 20818 catalog_manager.cc:5484] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:47.125072 25683 tablet_bootstrap.cc:654] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.126766 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83384d743f874d77bc8a12d1c9834736" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:47.127991 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83384d743f874d77bc8a12d1c9834736" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
W20240501 20:09:47.128562 23644 leader_election.cc:341] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:47.129141 23644 leader_election.cc:341] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:47.131384 25685 ts_tablet_manager.cc:1398] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 20:09:47.133164 25683 tablet_bootstrap.cc:492] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:47.133304 25683 ts_tablet_manager.cc:1367] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:09:47.134145 25683 raft_consensus.cc:348] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.134363 25683 raft_consensus.cc:374] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.134457 25683 raft_consensus.cc:729] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.134810 25683 consensus_queue.cc:260] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.135457 25683 ts_tablet_manager.cc:1398] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:47.155912 25532 raft_consensus.cc:482] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.156086 25532 raft_consensus.cc:504] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:47.156766 25532 leader_election.cc:288] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:47.157516 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaaf183dac384f179b7abbade02a8d03" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:47.157734 23844 raft_consensus.cc:2454] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:47.158147 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaaf183dac384f179b7abbade02a8d03" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:47.158360 23711 raft_consensus.cc:2454] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:47.158686 24046 leader_election.cc:302] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:47.159278 25532 raft_consensus.cc:2781] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:47.159391 25532 raft_consensus.cc:482] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:47.159477 25532 raft_consensus.cc:3037] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.166049 25532 raft_consensus.cc:504] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:47.167748 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaaf183dac384f179b7abbade02a8d03" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:47.168009 23844 raft_consensus.cc:3037] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.172016 23844 raft_consensus.cc:2454] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:47.172711 24046 leader_election.cc:302] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:47.174610 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaaf183dac384f179b7abbade02a8d03" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:47.174871 23711 raft_consensus.cc:3037] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.178925 23711 raft_consensus.cc:2454] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:47.179384 25532 leader_election.cc:288] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:47.180444 25532 raft_consensus.cc:2781] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:47.180903 25532 raft_consensus.cc:686] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:47.181703 25532 consensus_queue.cc:237] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:47.195144 20818 catalog_manager.cc:5484] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:47.220193 23710 consensus_queue.cc:237] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:47.222172 24108 raft_consensus.cc:1261] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:47.222308 21357 raft_consensus.cc:1261] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:47.222700 25701 consensus_queue.cc:1035] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:47.222973 25424 consensus_queue.cc:1035] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:47.225685 25701 raft_consensus.cc:2932] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:47.226394 24108 raft_consensus.cc:2932] T 939b60669869461483f50e61d8dc633b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:47.232352 20818 catalog_manager.cc:5484] T 939b60669869461483f50e61d8dc633b P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:47.239367 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet 939b60669869461483f50e61d8dc633b with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:47.239296 21357 raft_consensus.cc:2932] T 939b60669869461483f50e61d8dc633b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:47.239771 25606 tablet_replica.cc:330] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:47.240046 25148 raft_consensus.cc:482] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.240096 25606 raft_consensus.cc:2229] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.240221 25148 raft_consensus.cc:504] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.240365 25606 raft_consensus.cc:2258] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.240875 25148 leader_election.cc:288] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:47.241216 25606 ts_tablet_manager.cc:1875] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:47.242599 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f23f1dafeb4447b33cb7f5bdbf03c4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:47.242815 23710 raft_consensus.cc:2454] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:47.242879 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f23f1dafeb4447b33cb7f5bdbf03c4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:47.243108 23844 raft_consensus.cc:2454] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:47.243285 20902 leader_election.cc:302] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:47.243821 25148 raft_consensus.cc:2781] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:47.243923 25148 raft_consensus.cc:482] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:47.244029 25148 raft_consensus.cc:3037] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.246176 25410 raft_consensus.cc:482] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.246397 25410 raft_consensus.cc:504] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.247239 25410 leader_election.cc:288] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:47.247889 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f23f1dafeb4447b33cb7f5bdbf03c4" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:47.248138 23710 raft_consensus.cc:2454] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:47.248803 23781 leader_election.cc:302] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:47.249112 25410 raft_consensus.cc:2781] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:47.249219 25410 raft_consensus.cc:482] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:47.249323 25410 raft_consensus.cc:3037] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.255350 25148 raft_consensus.cc:504] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.255594 25606 ts_tablet_manager.cc:1888] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:09:47.255823 25606 log.cc:1196] T 939b60669869461483f50e61d8dc633b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/939b60669869461483f50e61d8dc633b
I20240501 20:09:47.256556 25410 raft_consensus.cc:504] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.257254 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 939b60669869461483f50e61d8dc633b (table rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]) successfully deleted
I20240501 20:09:47.257763 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f23f1dafeb4447b33cb7f5bdbf03c4" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:47.257982 23710 raft_consensus.cc:3037] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.258113 25148 leader_election.cc:288] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:47.258481 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f23f1dafeb4447b33cb7f5bdbf03c4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:47.258780 23844 raft_consensus.cc:2379] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cda19d6a81c0473db302535ce2ada7f0 in current term 1: Already voted for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in this term.
I20240501 20:09:47.259078 25410 leader_election.cc:288] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:47.259792 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f23f1dafeb4447b33cb7f5bdbf03c4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:47.260354 20902 leader_election.cc:302] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:47.260897 25148 raft_consensus.cc:2726] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:09:47.261305 23710 raft_consensus.cc:2454] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:47.261758 23781 leader_election.cc:302] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:47.262023 25410 raft_consensus.cc:2781] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:47.262226 25410 raft_consensus.cc:686] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:47.262496 25410 consensus_queue.cc:237] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:47.265635 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f23f1dafeb4447b33cb7f5bdbf03c4" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:47.265985 20967 raft_consensus.cc:2379] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate cda19d6a81c0473db302535ce2ada7f0 in this term.
I20240501 20:09:47.266631 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f23f1dafeb4447b33cb7f5bdbf03c4" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:47.266858 20967 raft_consensus.cc:2379] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate cda19d6a81c0473db302535ce2ada7f0 in this term.
I20240501 20:09:47.268083 20820 catalog_manager.cc:5484] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:47.275171 25148 raft_consensus.cc:482] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.275292 25148 raft_consensus.cc:504] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:47.275875 25148 leader_election.cc:288] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:47.276405 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718853c0da3d4a68ab478a6e79cdd438" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:47.276417 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718853c0da3d4a68ab478a6e79cdd438" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:47.276715 23976 raft_consensus.cc:2379] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cda19d6a81c0473db302535ce2ada7f0 in current term 1: Already voted for candidate 4af096f5a6a9440a94f6682b399efe20 in this term.
I20240501 20:09:47.277199 20900 leader_election.cc:302] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0; no voters: 4af096f5a6a9440a94f6682b399efe20, 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:47.277484 25148 raft_consensus.cc:3037] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.279676 25148 raft_consensus.cc:2726] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:47.346832 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:47.371775 23710 raft_consensus.cc:1261] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:47.371899 23844 raft_consensus.cc:1261] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:47.372879 25713 consensus_queue.cc:1035] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:47.373474 25713 consensus_queue.cc:1035] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:47.378351 25769 tablet.cc:1722] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:47.378777 20825 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet eaaf183dac384f179b7abbade02a8d03 (table rebalancer_dynamic_table [id=2ead7a0225924dc8ac72634a63383e36]). Expected version 1 got 0
I20240501 20:09:47.379443 25769 tablet.cc:1620] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:47.390872 25770 tablet.cc:1722] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:47.393385 25770 tablet.cc:1620] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:47.393667 20824 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet eaaf183dac384f179b7abbade02a8d03 (table rebalancer_dynamic_table [id=2ead7a0225924dc8ac72634a63383e36]). Expected version 1 got 0
I20240501 20:09:47.396211 20820 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=2ead7a0225924dc8ac72634a63383e36] alter complete (version 1)
I20240501 20:09:47.457056 20824 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet eaaf183dac384f179b7abbade02a8d03 (table rebalancer_dynamic_table [id=2ead7a0225924dc8ac72634a63383e36]). Expected version 1 got 0
I20240501 20:09:47.459980 25783 tablet.cc:1722] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:47.460680 25783 tablet.cc:1620] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:47.474570 20967 raft_consensus.cc:3037] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.479063 25701 consensus_queue.cc:1035] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:47.505517 20821 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:47.508463 25701 consensus_queue.cc:1035] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:47.518675 25688 consensus_queue.cc:1035] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:47.525693 25769 tablet.cc:1722] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:47.526242 25769 tablet.cc:1620] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:47.526329 21208 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:47.527316 25770 tablet.cc:1722] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:47.527798 25770 tablet.cc:1620] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:47.535688 23689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:47.539088 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=2ead7a0225924dc8ac72634a63383e36] alter complete (version 2)
I20240501 20:09:47.549553 25783 tablet.cc:1722] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:47.550109 25783 tablet.cc:1620] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:47.572289 25706 consensus_queue.cc:1035] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:47.658064 20821 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:47.663447 20945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:47.663935 21336 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:09:47.665751 23957 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:47.701792 25769 tablet.cc:1722] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:47.710920 25770 tablet.cc:1722] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:47.716849 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=2ead7a0225924dc8ac72634a63383e36] alter complete (version 3)
I20240501 20:09:47.718122 21076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:47.720731 25783 tablet.cc:1722] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:47.733727 25410 consensus_queue.cc:1035] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:47.741029 24088 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:47.741456 23824 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:47.764693 25460 consensus_queue.cc:1035] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:47.784889 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:47.797611 23957 tablet_service.cc:1467] Processing CreateTablet for tablet a163a3edf82e45f39b4f2a26f0466e9b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.798125 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a163a3edf82e45f39b4f2a26f0466e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.801633 23689 tablet_service.cc:1467] Processing CreateTablet for tablet eb612facbc174e0b957294af03ef2864 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.801716 24088 tablet_service.cc:1467] Processing CreateTablet for tablet eb612facbc174e0b957294af03ef2864 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.802073 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb612facbc174e0b957294af03ef2864. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.802181 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb612facbc174e0b957294af03ef2864. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.805344 23825 tablet_service.cc:1467] Processing CreateTablet for tablet b472171e77e144ffa8fa4cdbd40a03a0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.805766 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b472171e77e144ffa8fa4cdbd40a03a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.808012 25811 tablet_bootstrap.cc:492] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:47.807987 24089 tablet_service.cc:1467] Processing CreateTablet for tablet b472171e77e144ffa8fa4cdbd40a03a0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.808141 23824 tablet_service.cc:1467] Processing CreateTablet for tablet eb612facbc174e0b957294af03ef2864 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.808421 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b472171e77e144ffa8fa4cdbd40a03a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.808562 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb612facbc174e0b957294af03ef2864. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.809887 25811 tablet_bootstrap.cc:654] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.812764 21208 tablet_service.cc:1467] Processing CreateTablet for tablet b472171e77e144ffa8fa4cdbd40a03a0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.813213 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b472171e77e144ffa8fa4cdbd40a03a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.835969 20824 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:47.838611 20824 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:47.838833 23691 tablet_service.cc:1467] Processing CreateTablet for tablet a163a3edf82e45f39b4f2a26f0466e9b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.838877 24090 tablet_service.cc:1467] Processing CreateTablet for tablet a163a3edf82e45f39b4f2a26f0466e9b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.839320 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a163a3edf82e45f39b4f2a26f0466e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.839324 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a163a3edf82e45f39b4f2a26f0466e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.856972 20824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet eaaf183dac384f179b7abbade02a8d03
I20240501 20:09:47.859313 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet eaaf183dac384f179b7abbade02a8d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:47 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:47.868575 24090 tablet_service.cc:1514] Processing DeleteTablet for tablet eaaf183dac384f179b7abbade02a8d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:47 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:47.869717 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet eaaf183dac384f179b7abbade02a8d03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:47 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:47.875027 25817 tablet_replica.cc:330] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:47.875388 25817 raft_consensus.cc:2229] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.875726 25817 raft_consensus.cc:2258] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.879137 25815 tablet_bootstrap.cc:492] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:47.879776 25818 tablet_bootstrap.cc:492] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:47.880481 25820 tablet_replica.cc:330] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:47.879962 25817 ts_tablet_manager.cc:1875] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:47.878734 25814 tablet_bootstrap.cc:492] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:47.882874 25820 raft_consensus.cc:2229] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.883176 25820 raft_consensus.cc:2258] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.883559 25816 tablet_bootstrap.cc:492] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:47.884995 25819 tablet_replica.cc:330] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:47.885285 25819 raft_consensus.cc:2229] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:47.886035 25819 raft_consensus.cc:2258] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.886505 25820 ts_tablet_manager.cc:1875] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:47.888397 25815 tablet_bootstrap.cc:654] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.888442 25817 ts_tablet_manager.cc:1888] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:47.889972 25817 log.cc:1196] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/eaaf183dac384f179b7abbade02a8d03
I20240501 20:09:47.890028 25814 tablet_bootstrap.cc:654] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.890499 25817 ts_tablet_manager.cc:1909] T eaaf183dac384f179b7abbade02a8d03 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:47.890509 25816 tablet_bootstrap.cc:654] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.891844 25819 ts_tablet_manager.cc:1875] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:47.892340 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet eaaf183dac384f179b7abbade02a8d03 (table rebalancer_dynamic_table_new_name [id=2ead7a0225924dc8ac72634a63383e36]) successfully deleted
I20240501 20:09:47.882238 25818 tablet_bootstrap.cc:654] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.896514 25815 tablet_bootstrap.cc:492] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:47.899637 25811 tablet_bootstrap.cc:492] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:47.899844 25811 ts_tablet_manager.cc:1367] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.092s user 0.022s sys 0.035s
I20240501 20:09:47.899978 25818 tablet_bootstrap.cc:492] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:47.900182 25818 ts_tablet_manager.cc:1367] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20240501 20:09:47.900696 25811 raft_consensus.cc:348] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.900985 25811 raft_consensus.cc:374] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.900993 25818 raft_consensus.cc:348] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.901134 25811 raft_consensus.cc:729] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.901319 25818 raft_consensus.cc:374] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.901461 25818 raft_consensus.cc:729] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.901477 25811 consensus_queue.cc:260] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.901746 25818 consensus_queue.cc:260] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.903044 25811 ts_tablet_manager.cc:1398] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:47.903213 25818 ts_tablet_manager.cc:1398] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:47.903381 25811 tablet_bootstrap.cc:492] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:47.903498 25818 tablet_bootstrap.cc:492] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:47.904626 25713 raft_consensus.cc:482] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.904788 25713 raft_consensus.cc:504] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.905508 25713 leader_election.cc:288] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:47.907021 25811 tablet_bootstrap.cc:654] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.907438 25818 tablet_bootstrap.cc:654] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.909219 25820 ts_tablet_manager.cc:1888] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:47.909155 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb612facbc174e0b957294af03ef2864" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:47.909440 25820 log.cc:1196] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/eaaf183dac384f179b7abbade02a8d03
W20240501 20:09:47.909701 24046 leader_election.cc:341] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:47.910248 25820 ts_tablet_manager.cc:1909] T eaaf183dac384f179b7abbade02a8d03 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:09:47.910441 25819 ts_tablet_manager.cc:1888] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:47.910604 25819 log.cc:1196] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/eaaf183dac384f179b7abbade02a8d03
I20240501 20:09:47.912029 25819 ts_tablet_manager.cc:1909] T eaaf183dac384f179b7abbade02a8d03 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:47.912971 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet eaaf183dac384f179b7abbade02a8d03 (table rebalancer_dynamic_table_new_name [id=2ead7a0225924dc8ac72634a63383e36]) successfully deleted
I20240501 20:09:47.913592 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet eaaf183dac384f179b7abbade02a8d03 (table rebalancer_dynamic_table_new_name [id=2ead7a0225924dc8ac72634a63383e36]) successfully deleted
I20240501 20:09:47.917831 25814 tablet_bootstrap.cc:492] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:47.906255 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb612facbc174e0b957294af03ef2864" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:47.918668 25815 ts_tablet_manager.cc:1367] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
W20240501 20:09:47.918864 24045 leader_election.cc:341] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:47.919075 24045 leader_election.cc:302] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:47.919595 25713 raft_consensus.cc:2726] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:47.919931 25814 ts_tablet_manager.cc:1367] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20240501 20:09:47.921514 25815 raft_consensus.cc:348] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.921906 25815 raft_consensus.cc:374] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.922068 25815 raft_consensus.cc:729] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.922436 25815 consensus_queue.cc:260] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.921226 25814 raft_consensus.cc:348] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.927338 25815 ts_tablet_manager.cc:1398] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:09:47.927594 25814 raft_consensus.cc:374] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.927815 25814 raft_consensus.cc:729] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.929867 25816 tablet_bootstrap.cc:492] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:47.930153 25816 ts_tablet_manager.cc:1367] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.003s
I20240501 20:09:47.939684 25816 raft_consensus.cc:348] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:47.942735 25816 raft_consensus.cc:374] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.942919 25816 raft_consensus.cc:729] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.950361 25811 tablet_bootstrap.cc:492] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:47.950531 25811 ts_tablet_manager.cc:1367] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.047s user 0.019s sys 0.017s
I20240501 20:09:47.951285 25815 tablet_bootstrap.cc:492] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:47.951206 25811 raft_consensus.cc:348] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.951414 25811 raft_consensus.cc:374] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.951526 25811 raft_consensus.cc:729] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.951807 25811 consensus_queue.cc:260] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.943328 25816 consensus_queue.cc:260] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:47.928285 25814 consensus_queue.cc:260] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.953416 25814 ts_tablet_manager.cc:1398] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240501 20:09:47.954475 25811 ts_tablet_manager.cc:1398] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:09:47.954783 25811 tablet_bootstrap.cc:492] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:47.956784 25811 tablet_bootstrap.cc:654] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.956813 25816 ts_tablet_manager.cc:1398] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240501 20:09:47.956552 25706 raft_consensus.cc:482] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.959242 25706 raft_consensus.cc:504] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.960017 25706 leader_election.cc:288] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:47.960728 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b472171e77e144ffa8fa4cdbd40a03a0" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:47.960963 23844 raft_consensus.cc:2454] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:47.961447 21164 leader_election.cc:302] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:47.962204 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b472171e77e144ffa8fa4cdbd40a03a0" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:47.962419 24108 raft_consensus.cc:2454] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:47.963100 25706 raft_consensus.cc:2781] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:47.963191 25706 raft_consensus.cc:482] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:47.963279 25706 raft_consensus.cc:3037] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.966074 25706 raft_consensus.cc:504] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.967167 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b472171e77e144ffa8fa4cdbd40a03a0" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:47.967365 23844 raft_consensus.cc:3037] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.970068 23844 raft_consensus.cc:2454] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:47.970479 21164 leader_election.cc:302] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:47.970817 25074 raft_consensus.cc:2781] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:47.970940 25074 raft_consensus.cc:686] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:47.971185 25074 consensus_queue.cc:237] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.972368 25706 leader_election.cc:288] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:47.973186 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b472171e77e144ffa8fa4cdbd40a03a0" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:47.973405 24108 raft_consensus.cc:3037] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.978156 24108 raft_consensus.cc:2454] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:47.991362 25818 tablet_bootstrap.cc:492] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:47.991580 25818 ts_tablet_manager.cc:1367] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.088s user 0.029s sys 0.023s
I20240501 20:09:47.992424 25818 raft_consensus.cc:348] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.992653 25818 raft_consensus.cc:374] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.992722 25818 raft_consensus.cc:729] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.992949 25818 consensus_queue.cc:260] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:47.993170 20818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:47.993444 25818 ts_tablet_manager.cc:1398] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:48.000963 25811 tablet_bootstrap.cc:492] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:48.001179 25811 ts_tablet_manager.cc:1367] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.015s
I20240501 20:09:48.002130 25811 raft_consensus.cc:348] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.002328 25811 raft_consensus.cc:374] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.002409 25811 raft_consensus.cc:729] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.002652 25811 consensus_queue.cc:260] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.009101 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f0c604dc1284b139dd41080a6a2ef4c]), partition=
I20240501 20:09:48.009447 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f0c604dc1284b139dd41080a6a2ef4c]), partition=
I20240501 20:09:48.009523 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b0b77490e74e5b9643a9fe8ff2eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.009829 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b0b77490e74e5b9643a9fe8ff2eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.012022 24090 tablet_service.cc:1467] Processing CreateTablet for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f0c604dc1284b139dd41080a6a2ef4c]), partition=
I20240501 20:09:48.012372 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b0b77490e74e5b9643a9fe8ff2eec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.012998 25811 ts_tablet_manager.cc:1398] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20240501 20:09:48.021798 25811 tablet_bootstrap.cc:492] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:48.023366 25701 raft_consensus.cc:482] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.023460 25811 tablet_bootstrap.cc:654] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.023505 25701 raft_consensus.cc:504] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:48.024194 25701 leader_election.cc:288] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:48.024761 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb612facbc174e0b957294af03ef2864" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:48.025012 23846 raft_consensus.cc:2454] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:48.025758 23647 leader_election.cc:302] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:48.026475 25701 raft_consensus.cc:2781] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:48.026585 25701 raft_consensus.cc:482] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:48.026679 25701 raft_consensus.cc:3037] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.029493 25701 raft_consensus.cc:504] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:48.030918 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb612facbc174e0b957294af03ef2864" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:48.031178 24110 raft_consensus.cc:3037] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.033972 24110 raft_consensus.cc:2454] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:48.034876 23646 leader_election.cc:302] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:48.037717 25836 raft_consensus.cc:2781] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:48.037788 25701 leader_election.cc:288] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:48.038144 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb612facbc174e0b957294af03ef2864" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:48.038336 24108 raft_consensus.cc:2362] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1. Re-sending same reply.
I20240501 20:09:48.040832 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb612facbc174e0b957294af03ef2864" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:48.041119 23846 raft_consensus.cc:3037] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.043856 25836 raft_consensus.cc:686] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:48.044678 25836 consensus_queue.cc:237] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:48.047443 25811 tablet_bootstrap.cc:492] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:48.047659 25811 ts_tablet_manager.cc:1367] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20240501 20:09:48.048399 25811 raft_consensus.cc:348] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.048604 25811 raft_consensus.cc:374] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.048698 25811 raft_consensus.cc:729] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.048902 23846 raft_consensus.cc:2454] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:48.048969 25811 consensus_queue.cc:260] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.050828 25811 ts_tablet_manager.cc:1398] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:48.051928 25834 tablet_bootstrap.cc:492] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:48.056079 25834 tablet_bootstrap.cc:654] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.061352 25713 raft_consensus.cc:482] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.061532 25713 raft_consensus.cc:504] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.062361 25713 leader_election.cc:288] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:48.063220 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163a3edf82e45f39b4f2a26f0466e9b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:48.064281 23976 raft_consensus.cc:2454] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:48.064955 24043 leader_election.cc:302] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:48.065393 25713 raft_consensus.cc:2781] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:48.065521 25713 raft_consensus.cc:482] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:48.065616 25713 raft_consensus.cc:3037] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.068282 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163a3edf82e45f39b4f2a26f0466e9b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
W20240501 20:09:48.069039 24045 leader_election.cc:341] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:48.073633 25713 raft_consensus.cc:504] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.075086 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163a3edf82e45f39b4f2a26f0466e9b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:48.075410 23976 raft_consensus.cc:3037] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.079155 25834 tablet_bootstrap.cc:492] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:48.079372 23976 raft_consensus.cc:2454] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:48.079464 25834 ts_tablet_manager.cc:1367] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240501 20:09:48.079919 24043 leader_election.cc:302] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:48.080304 25532 raft_consensus.cc:2781] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:48.080660 25834 raft_consensus.cc:348] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.080943 25834 raft_consensus.cc:374] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.081085 25834 raft_consensus.cc:729] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.081421 25834 consensus_queue.cc:260] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.082183 25834 ts_tablet_manager.cc:1398] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:48.082542 20818 catalog_manager.cc:5484] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:48.086969 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163a3edf82e45f39b4f2a26f0466e9b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
W20240501 20:09:48.089193 24045 leader_election.cc:341] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:48.091892 25713 leader_election.cc:288] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:48.096367 25815 tablet_bootstrap.cc:700] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent opening tablet: real 0.107s user 0.001s sys 0.000s
I20240501 20:09:48.108621 25815 tablet_bootstrap.cc:654] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.110962 25532 raft_consensus.cc:686] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:48.111312 25532 consensus_queue.cc:237] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.113899 20818 catalog_manager.cc:5484] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:48.114532 25815 tablet_bootstrap.cc:492] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:48.114866 25815 ts_tablet_manager.cc:1367] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.164s user 0.005s sys 0.000s
I20240501 20:09:48.116051 25815 raft_consensus.cc:348] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.116362 25815 raft_consensus.cc:374] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.116472 25815 raft_consensus.cc:729] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.116797 25815 consensus_queue.cc:260] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.118665 25815 ts_tablet_manager.cc:1398] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:09:48.118995 25815 tablet_bootstrap.cc:492] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:48.120637 25815 tablet_bootstrap.cc:654] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.122625 25815 tablet_bootstrap.cc:492] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:48.122828 25815 ts_tablet_manager.cc:1367] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 20:09:48.123562 25815 raft_consensus.cc:348] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.123833 25815 raft_consensus.cc:374] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.123960 25815 raft_consensus.cc:729] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.124264 25815 consensus_queue.cc:260] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.124850 25815 ts_tablet_manager.cc:1398] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:48.126264 20821 catalog_manager.cc:5484] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:48.153635 25785 raft_consensus.cc:482] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.153818 25785 raft_consensus.cc:504] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.154652 25785 leader_election.cc:288] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:48.155139 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b0b77490e74e5b9643a9fe8ff2eec6" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:48.155365 24108 raft_consensus.cc:2454] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:48.155622 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b0b77490e74e5b9643a9fe8ff2eec6" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:48.155915 23711 raft_consensus.cc:2454] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:48.156383 21033 leader_election.cc:302] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:48.156661 25785 raft_consensus.cc:2781] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:48.156774 25785 raft_consensus.cc:482] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:48.156885 25785 raft_consensus.cc:3037] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.157135 23711 consensus_queue.cc:237] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:48.159251 23846 raft_consensus.cc:1261] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:48.159322 21225 raft_consensus.cc:1261] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:48.159857 25837 consensus_queue.cc:1035] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:48.160141 25836 consensus_queue.cc:1035] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:48.160321 25785 raft_consensus.cc:504] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.160916 25785 leader_election.cc:288] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:48.161427 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b0b77490e74e5b9643a9fe8ff2eec6" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:48.161446 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b0b77490e74e5b9643a9fe8ff2eec6" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:48.161623 23711 raft_consensus.cc:3037] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.161671 24108 raft_consensus.cc:3037] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.165179 23711 raft_consensus.cc:2454] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:48.165287 24108 raft_consensus.cc:2454] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:48.165659 21033 leader_election.cc:302] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:48.165971 25785 raft_consensus.cc:2781] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:48.166107 25785 raft_consensus.cc:686] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:48.166401 25785 consensus_queue.cc:237] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.172480 25836 raft_consensus.cc:2932] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:48.173101 23846 raft_consensus.cc:2932] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:48.174293 20818 catalog_manager.cc:5484] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:48.179180 21225 raft_consensus.cc:2932] T 4715a768081a49228ab47b8071c40c8f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:48.179750 20821 catalog_manager.cc:5484] T 4715a768081a49228ab47b8071c40c8f P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:48.187526 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 4715a768081a49228ab47b8071c40c8f with delete_type TABLET_DATA_TOMBSTONED (TS cda19d6a81c0473db302535ce2ada7f0 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:48.191992 25859 tablet_replica.cc:330] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:48.192257 25859 raft_consensus.cc:2229] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:48.192534 25859 raft_consensus.cc:2258] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:48.193243 25859 ts_tablet_manager.cc:1875] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:48.199891 25859 ts_tablet_manager.cc:1888] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:09:48.200233 25859 log.cc:1196] T 4715a768081a49228ab47b8071c40c8f P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/4715a768081a49228ab47b8071c40c8f
I20240501 20:09:48.201216 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 4715a768081a49228ab47b8071c40c8f (table rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]) successfully deleted
I20240501 20:09:48.289475 25836 raft_consensus.cc:482] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.289677 25836 raft_consensus.cc:504] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:48.290433 25836 leader_election.cc:288] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:48.290860 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163a3edf82e45f39b4f2a26f0466e9b" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:48.291144 23976 raft_consensus.cc:2379] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51bf640e895b4e5eb91bdf7509ea7e46 in current term 1: Already voted for candidate 8559229e044b4e1ea661b881b3f9af50 in this term.
I20240501 20:09:48.291466 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163a3edf82e45f39b4f2a26f0466e9b" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:48.293114 23646 leader_election.cc:302] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46; no voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:48.293514 25836 raft_consensus.cc:3037] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.296456 25836 raft_consensus.cc:2726] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:48.305683 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:48.354777 23711 raft_consensus.cc:1261] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:48.355037 24108 raft_consensus.cc:1261] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:48.355437 25785 consensus_queue.cc:1035] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.356613 25785 consensus_queue.cc:1035] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.372040 20818 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (table rebalancer_dynamic_table [id=7f0c604dc1284b139dd41080a6a2ef4c]). Expected version 1 got 0
I20240501 20:09:48.373842 20818 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (table rebalancer_dynamic_table [id=7f0c604dc1284b139dd41080a6a2ef4c]). Expected version 1 got 0
I20240501 20:09:48.377313 25905 tablet.cc:1722] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:48.378696 25905 tablet.cc:1620] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: MemRowSet was empty: no flush needed.
I20240501 20:09:48.383730 25906 tablet.cc:1722] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:48.386310 25906 tablet.cc:1620] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:48.386914 20818 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (table rebalancer_dynamic_table [id=7f0c604dc1284b139dd41080a6a2ef4c]). Expected version 1 got 0
I20240501 20:09:48.397125 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=7f0c604dc1284b139dd41080a6a2ef4c] alter complete (version 1)
I20240501 20:09:48.417160 25907 tablet.cc:1722] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:48.420069 25688 consensus_queue.cc:1035] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.422436 20818 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (table rebalancer_dynamic_table [id=7f0c604dc1284b139dd41080a6a2ef4c]). Expected version 1 got 0
I20240501 20:09:48.429147 25907 tablet.cc:1620] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:48.434556 25836 consensus_queue.cc:1035] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.455026 21208 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:48.459846 23689 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:48.488950 23824 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:48.495432 23957 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:48.501987 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:48.515507 25906 tablet.cc:1722] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:48.516131 25906 tablet.cc:1620] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:48.516227 25905 tablet.cc:1722] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:48.516692 25905 tablet.cc:1620] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: MemRowSet was empty: no flush needed.
I20240501 20:09:48.516904 25074 consensus_queue.cc:1035] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.527626 25907 tablet.cc:1722] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:48.528173 25907 tablet.cc:1620] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:48.530201 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=7f0c604dc1284b139dd41080a6a2ef4c] alter complete (version 2)
I20240501 20:09:48.537205 25836 consensus_queue.cc:1035] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.559973 21336 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:09:48.572372 25532 consensus_queue.cc:1035] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.585836 20945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:48.607771 21076 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:48.607980 25713 consensus_queue.cc:1035] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.635035 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:48.640655 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:48.641923 20820 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (table rebalancer_dynamic_table_new_name [id=7f0c604dc1284b139dd41080a6a2ef4c]). Expected version 3 got 2
I20240501 20:09:48.646531 24090 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:48.659353 24090 tablet_service.cc:1467] Processing CreateTablet for tablet d3c0aa45580749a4bd6fa7ccd635cb38 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.659847 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c0aa45580749a4bd6fa7ccd635cb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.665745 23957 tablet_service.cc:1467] Processing CreateTablet for tablet d3c0aa45580749a4bd6fa7ccd635cb38 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.666194 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c0aa45580749a4bd6fa7ccd635cb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.672253 21208 tablet_service.cc:1467] Processing CreateTablet for tablet d3c0aa45580749a4bd6fa7ccd635cb38 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.672681 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c0aa45580749a4bd6fa7ccd635cb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.678381 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbbd5fb15a64e0b8495e63766c49a57 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.678840 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbbd5fb15a64e0b8495e63766c49a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.679055 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbbd5fb15a64e0b8495e63766c49a57 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.679459 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbbd5fb15a64e0b8495e63766c49a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.690084 25934 tablet_bootstrap.cc:492] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:48.690531 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 6c31dffb0482438ebbc6243ad54752e9 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.690951 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c31dffb0482438ebbc6243ad54752e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.694510 25933 tablet_bootstrap.cc:492] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:48.695348 25934 tablet_bootstrap.cc:654] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.695945 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 6c31dffb0482438ebbc6243ad54752e9 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.696374 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c31dffb0482438ebbc6243ad54752e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.698118 25933 tablet_bootstrap.cc:654] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.699529 25937 tablet_bootstrap.cc:492] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:48.700037 25935 tablet_bootstrap.cc:492] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:48.701305 25905 tablet.cc:1722] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:48.701831 25935 tablet_bootstrap.cc:654] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.705986 25937 tablet_bootstrap.cc:654] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.715343 25906 tablet.cc:1722] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:48.716099 25907 tablet.cc:1722] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:48.717217 25934 tablet_bootstrap.cc:492] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:48.717401 25934 ts_tablet_manager.cc:1367] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.005s
I20240501 20:09:48.718333 25934 raft_consensus.cc:348] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:48.718575 25934 raft_consensus.cc:374] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.718675 25934 raft_consensus.cc:729] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.718995 25934 consensus_queue.cc:260] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:48.719636 25934 ts_tablet_manager.cc:1398] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 20:09:48.730610 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=7f0c604dc1284b139dd41080a6a2ef4c] alter complete (version 3)
I20240501 20:09:48.736557 25933 tablet_bootstrap.cc:492] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:48.736812 25933 ts_tablet_manager.cc:1367] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.012s
I20240501 20:09:48.737735 25933 raft_consensus.cc:348] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.738041 25933 raft_consensus.cc:374] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.738516 25933 raft_consensus.cc:729] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.738981 25933 consensus_queue.cc:260] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.741020 25933 ts_tablet_manager.cc:1398] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:09:48.741420 25933 tablet_bootstrap.cc:492] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:48.743422 25933 tablet_bootstrap.cc:654] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.744357 25937 tablet_bootstrap.cc:492] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:48.744608 25937 ts_tablet_manager.cc:1367] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.006s
I20240501 20:09:48.745630 25937 raft_consensus.cc:348] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:48.751719 25837 raft_consensus.cc:482] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.751905 25837 raft_consensus.cc:504] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.752645 25837 leader_election.cc:288] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:48.747028 25937 raft_consensus.cc:374] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.755034 25937 raft_consensus.cc:729] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.755688 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbbd5fb15a64e0b8495e63766c49a57" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
W20240501 20:09:48.756595 23645 leader_election.cc:341] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:48.757045 25937 consensus_queue.cc:260] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:48.759694 25935 tablet_bootstrap.cc:492] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:48.761315 25937 ts_tablet_manager.cc:1398] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240501 20:09:48.761345 25935 ts_tablet_manager.cc:1367] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20240501 20:09:48.770766 25935 raft_consensus.cc:348] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.771347 25939 tablet_bootstrap.cc:492] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:48.771759 25935 raft_consensus.cc:374] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.771871 25935 raft_consensus.cc:729] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.772130 25935 consensus_queue.cc:260] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.772939 25935 ts_tablet_manager.cc:1398] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20240501 20:09:48.778726 23647 leader_election.cc:341] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Not found: Tablet not found: 4bbbd5fb15a64e0b8495e63766c49a57
I20240501 20:09:48.778926 23647 leader_election.cc:302] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46; no voters: 61380023001a4ee9ada9ec2c6c1b7159, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:48.779409 25837 raft_consensus.cc:2726] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:48.780711 25933 tablet_bootstrap.cc:492] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:48.783691 25933 ts_tablet_manager.cc:1367] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.003s
I20240501 20:09:48.785998 25933 raft_consensus.cc:348] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:48.786275 25933 raft_consensus.cc:374] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.786351 25933 raft_consensus.cc:729] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.786614 25933 consensus_queue.cc:260] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:48.793421 25933 ts_tablet_manager.cc:1398] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
W20240501 20:09:48.800969 23779 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 223652us
I20240501 20:09:48.818233 24171 raft_consensus.cc:482] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.818459 24171 raft_consensus.cc:504] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.819308 24171 leader_election.cc:288] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:48.819840 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbbd5fb15a64e0b8495e63766c49a57 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.820336 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbbd5fb15a64e0b8495e63766c49a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.820438 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbbd5fb15a64e0b8495e63766c49a57" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:48.820652 23711 raft_consensus.cc:2454] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:48.821097 21293 leader_election.cc:302] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:09:48.822480 24171 raft_consensus.cc:2781] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:48.822595 24171 raft_consensus.cc:482] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:48.822723 24171 raft_consensus.cc:3037] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.826938 24171 raft_consensus.cc:504] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.828344 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbbd5fb15a64e0b8495e63766c49a57" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
W20240501 20:09:48.828898 21294 leader_election.cc:341] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:48.829138 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbbd5fb15a64e0b8495e63766c49a57" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:09:48.829571 21294 leader_election.cc:341] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:48.830140 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbbd5fb15a64e0b8495e63766c49a57" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:48.830392 23711 raft_consensus.cc:3037] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.831020 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 6c31dffb0482438ebbc6243ad54752e9 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.831473 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c31dffb0482438ebbc6243ad54752e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.831526 25939 tablet_bootstrap.cc:654] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.833307 23711 raft_consensus.cc:2454] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:48.834038 21293 leader_election.cc:302] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 61380023001a4ee9ada9ec2c6c1b7159; no voters: c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:48.834437 24565 raft_consensus.cc:2781] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:48.837044 24171 leader_election.cc:288] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:48.849946 20818 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:48.854499 24565 raft_consensus.cc:686] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:48.855816 24565 consensus_queue.cc:237] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.856209 20818 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:48.860260 20818 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 03b0b77490e74e5b9643a9fe8ff2eec6
I20240501 20:09:48.860513 25947 tablet_bootstrap.cc:492] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:48.861021 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:48 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:48.867465 25951 tablet_replica.cc:330] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:48.868074 25951 raft_consensus.cc:2229] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:48.868152 25713 raft_consensus.cc:482] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.868340 25713 raft_consensus.cc:504] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:48.868409 20823 catalog_manager.cc:5484] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:48.868971 25713 leader_election.cc:288] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:48.869386 25951 raft_consensus.cc:2258] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:48.869632 24090 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:48 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:48.869714 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c0aa45580749a4bd6fa7ccd635cb38" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:48.869911 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c0aa45580749a4bd6fa7ccd635cb38" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:48.870206 21225 raft_consensus.cc:2454] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:48.870591 25951 ts_tablet_manager.cc:1875] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:48.870855 24046 leader_election.cc:302] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:48.871203 25713 raft_consensus.cc:2781] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:48.871304 25713 raft_consensus.cc:482] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:48.871451 25713 raft_consensus.cc:3037] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.871816 25939 tablet_bootstrap.cc:492] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:48.872284 25939 ts_tablet_manager.cc:1367] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.003s
I20240501 20:09:48.873404 25939 raft_consensus.cc:348] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:48.874464 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b0b77490e74e5b9643a9fe8ff2eec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:48 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:48.874460 25713 raft_consensus.cc:504] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:48.874720 25939 raft_consensus.cc:374] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.874928 25939 raft_consensus.cc:729] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
W20240501 20:09:48.875312 24043 leader_election.cc:341] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:48.875370 25939 consensus_queue.cc:260] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:48.876446 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c0aa45580749a4bd6fa7ccd635cb38" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:48.876657 21225 raft_consensus.cc:3037] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.877439 25947 tablet_bootstrap.cc:654] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.877602 25939 ts_tablet_manager.cc:1398] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:09:48.878374 25939 tablet_bootstrap.cc:492] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:48.879683 21225 raft_consensus.cc:2454] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:48.880257 24046 leader_election.cc:302] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:48.880663 25840 raft_consensus.cc:2781] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:48.880825 25840 raft_consensus.cc:686] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:48.881147 25840 consensus_queue.cc:237] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:48.883484 25939 tablet_bootstrap.cc:654] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.883529 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c0aa45580749a4bd6fa7ccd635cb38" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:48.883903 23977 raft_consensus.cc:3037] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.887184 23977 raft_consensus.cc:2454] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:48.889393 25713 leader_election.cc:288] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:48.891767 20818 catalog_manager.cc:5484] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:48.895917 25952 tablet_replica.cc:330] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:48.897055 25939 tablet_bootstrap.cc:492] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:48.897302 25939 ts_tablet_manager.cc:1367] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20240501 20:09:48.898090 25939 raft_consensus.cc:348] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:48.898398 25939 raft_consensus.cc:374] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.898510 25939 raft_consensus.cc:729] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.898805 25939 consensus_queue.cc:260] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:48.899919 25939 ts_tablet_manager.cc:1398] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:48.900261 25952 raft_consensus.cc:2229] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:48.900473 25952 raft_consensus.cc:2258] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:48.905117 25952 ts_tablet_manager.cc:1875] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:48.913038 25953 tablet_replica.cc:330] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:48.915382 25951 ts_tablet_manager.cc:1888] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:48.915803 25953 raft_consensus.cc:2229] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:48.916138 25953 raft_consensus.cc:2258] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:48.919514 25947 tablet_bootstrap.cc:492] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:48.919811 25947 ts_tablet_manager.cc:1367] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.007s
I20240501 20:09:48.920581 25947 raft_consensus.cc:348] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.920853 25947 raft_consensus.cc:374] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.920984 25947 raft_consensus.cc:729] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.921250 25947 consensus_queue.cc:260] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.922845 25947 ts_tablet_manager.cc:1398] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 20:09:48.923153 25947 tablet_bootstrap.cc:492] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:48.926488 25947 tablet_bootstrap.cc:654] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.926674 25951 log.cc:1196] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/03b0b77490e74e5b9643a9fe8ff2eec6
I20240501 20:09:48.927179 25953 ts_tablet_manager.cc:1875] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:48.929013 25951 ts_tablet_manager.cc:1909] T 03b0b77490e74e5b9643a9fe8ff2eec6 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:48.930760 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (table rebalancer_dynamic_table_new_name [id=7f0c604dc1284b139dd41080a6a2ef4c]) successfully deleted
I20240501 20:09:48.949334 25953 ts_tablet_manager.cc:1888] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:48.949957 25952 ts_tablet_manager.cc:1888] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:48.950163 25952 log.cc:1196] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/03b0b77490e74e5b9643a9fe8ff2eec6
I20240501 20:09:48.950616 25952 ts_tablet_manager.cc:1909] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:48.951519 25947 tablet_bootstrap.cc:492] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:48.951787 25947 ts_tablet_manager.cc:1367] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.003s
I20240501 20:09:48.952597 25947 raft_consensus.cc:348] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:48.952885 25947 raft_consensus.cc:374] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.953007 25947 raft_consensus.cc:729] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.953330 25947 consensus_queue.cc:260] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:48.954020 25947 ts_tablet_manager.cc:1398] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:48.955257 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (table rebalancer_dynamic_table_new_name [id=7f0c604dc1284b139dd41080a6a2ef4c]) successfully deleted
I20240501 20:09:48.958073 25953 log.cc:1196] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/03b0b77490e74e5b9643a9fe8ff2eec6
I20240501 20:09:48.958632 25953 ts_tablet_manager.cc:1909] T 03b0b77490e74e5b9643a9fe8ff2eec6 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:48.959908 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 03b0b77490e74e5b9643a9fe8ff2eec6 (table rebalancer_dynamic_table_new_name [id=7f0c604dc1284b139dd41080a6a2ef4c]) successfully deleted
I20240501 20:09:48.976459 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:48.978107 25946 raft_consensus.cc:482] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.978299 25946 raft_consensus.cc:504] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:48.979092 25946 leader_election.cc:288] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:48.979810 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbbd5fb15a64e0b8495e63766c49a57" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:48.979810 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbbd5fb15a64e0b8495e63766c49a57" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:48.980103 23711 raft_consensus.cc:2379] T 4bbbd5fb15a64e0b8495e63766c49a57 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate 61380023001a4ee9ada9ec2c6c1b7159 in this term.
I20240501 20:09:48.980585 23780 leader_election.cc:302] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:09:48.981287 25945 raft_consensus.cc:3037] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.985615 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8ac071609c4703b06265531ddc15ab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c]), partition=
I20240501 20:09:48.985996 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8ac071609c4703b06265531ddc15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.986151 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8ac071609c4703b06265531ddc15ab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c]), partition=
I20240501 20:09:48.986474 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8ac071609c4703b06265531ddc15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.987973 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8ac071609c4703b06265531ddc15ab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c]), partition=
I20240501 20:09:48.988302 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8ac071609c4703b06265531ddc15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.992353 25933 tablet_bootstrap.cc:492] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:48.993849 25933 tablet_bootstrap.cc:654] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.000146 25945 raft_consensus.cc:2726] T 4bbbd5fb15a64e0b8495e63766c49a57 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:49.001580 23976 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8"
dest_uuid: "744c5a532e724391830c0785d32eb2fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50794
I20240501 20:09:49.001840 23976 raft_consensus.cc:595] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:49.011576 25939 tablet_bootstrap.cc:492] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:49.013311 25939 tablet_bootstrap.cc:654] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.014753 25934 tablet_bootstrap.cc:492] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:49.017910 25934 tablet_bootstrap.cc:654] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.038086 25933 tablet_bootstrap.cc:492] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:49.038313 25933 ts_tablet_manager.cc:1367] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.020s
I20240501 20:09:49.038780 25574 raft_consensus.cc:979] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: : Instructing follower 8559229e044b4e1ea661b881b3f9af50 to start an election
I20240501 20:09:49.039065 25574 raft_consensus.cc:1067] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Signalling peer 8559229e044b4e1ea661b881b3f9af50 to start an election
I20240501 20:09:49.039602 25933 raft_consensus.cc:348] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:49.039880 25933 raft_consensus.cc:374] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.039979 25933 raft_consensus.cc:729] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.039958 24108 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8"
dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
from {username='slave'} at 127.13.251.135:52269
I20240501 20:09:49.040239 24108 raft_consensus.cc:482] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:49.040349 24108 raft_consensus.cc:3037] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.040324 25933 consensus_queue.cc:260] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:49.043196 24108 raft_consensus.cc:504] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:49.044072 25933 ts_tablet_manager.cc:1398] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:09:49.044256 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:49.044466 23711 raft_consensus.cc:3037] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.047336 23711 raft_consensus.cc:2454] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 2.
I20240501 20:09:49.049188 24108 leader_election.cc:288] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:49.050772 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:49.051079 21095 raft_consensus.cc:3037] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.054911 21095 raft_consensus.cc:2454] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 2.
I20240501 20:09:49.055999 24043 leader_election.cc:302] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:49.056433 25713 raft_consensus.cc:2781] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:49.056629 25713 raft_consensus.cc:686] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 2 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:49.056995 25713 consensus_queue.cc:237] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:49.059890 23711 raft_consensus.cc:1226] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Rejecting Update request from peer 744c5a532e724391830c0785d32eb2fb for earlier term 1. Current term is 2. Ops: []
I20240501 20:09:49.060564 25968 consensus_queue.cc:1046] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:49.060964 25574 raft_consensus.cc:3032] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:49.061048 25574 raft_consensus.cc:729] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:49.061373 25574 consensus_queue.cc:260] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:49.061827 25574 raft_consensus.cc:3037] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.065800 25934 tablet_bootstrap.cc:492] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:49.066045 25934 ts_tablet_manager.cc:1367] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.005s
I20240501 20:09:49.066872 25934 raft_consensus.cc:348] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:49.067478 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30eb2ed6c8b64204b4c3a11b41e53ec8" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:49.067821 25934 raft_consensus.cc:374] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.068423 25934 raft_consensus.cc:729] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.068779 25934 consensus_queue.cc:260] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:49.074872 25934 ts_tablet_manager.cc:1398] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 20:09:49.077169 25939 tablet_bootstrap.cc:492] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:49.077376 25939 ts_tablet_manager.cc:1367] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.010s
I20240501 20:09:49.078155 25939 raft_consensus.cc:348] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:49.078433 25939 raft_consensus.cc:374] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.078588 25939 raft_consensus.cc:729] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.078524 23976 raft_consensus.cc:2454] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 2.
I20240501 20:09:49.078908 25939 consensus_queue.cc:260] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:49.079586 25939 ts_tablet_manager.cc:1398] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:49.086120 20823 catalog_manager.cc:5484] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 1 to 2, leader changed from 744c5a532e724391830c0785d32eb2fb (127.13.251.135) to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 2 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:49.092294 25946 raft_consensus.cc:482] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:49.092512 25946 raft_consensus.cc:504] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.093288 25946 leader_election.cc:288] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:49.094287 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c31dffb0482438ebbc6243ad54752e9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:49.094545 23976 raft_consensus.cc:2454] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:49.095438 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c31dffb0482438ebbc6243ad54752e9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:49.095762 23711 raft_consensus.cc:2454] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:49.096467 23779 leader_election.cc:302] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:49.097736 25945 raft_consensus.cc:2781] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:49.097856 25945 raft_consensus.cc:482] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:49.097977 25945 raft_consensus.cc:3037] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:49.101888 25945 raft_consensus.cc:504] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.105810 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c31dffb0482438ebbc6243ad54752e9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:49.106084 23711 raft_consensus.cc:3037] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:49.110064 23711 raft_consensus.cc:2454] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:49.110685 23781 leader_election.cc:302] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:49.111092 25946 raft_consensus.cc:2781] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:49.111256 25946 raft_consensus.cc:686] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:49.111572 25946 consensus_queue.cc:237] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.112742 25945 leader_election.cc:288] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:49.114269 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c31dffb0482438ebbc6243ad54752e9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:49.114478 23976 raft_consensus.cc:3037] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:49.115473 20823 catalog_manager.cc:5484] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:49.118068 23976 raft_consensus.cc:2454] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:49.180910 25968 raft_consensus.cc:482] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:49.181206 25968 raft_consensus.cc:504] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:49.182262 25968 leader_election.cc:288] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:49.182613 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8ac071609c4703b06265531ddc15ab" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:49.182860 23711 raft_consensus.cc:2454] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:49.183959 23911 leader_election.cc:302] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:49.184365 25968 raft_consensus.cc:2781] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:49.184489 25968 raft_consensus.cc:482] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:49.184607 25968 raft_consensus.cc:3037] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:49.189705 25968 raft_consensus.cc:504] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:49.190908 25968 leader_election.cc:288] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:49.190868 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8ac071609c4703b06265531ddc15ab" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:49.191066 23711 raft_consensus.cc:3037] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:49.197263 23711 raft_consensus.cc:2454] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:49.197685 23911 leader_election.cc:302] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:49.198074 25574 raft_consensus.cc:2781] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:49.202484 25574 raft_consensus.cc:686] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:49.202853 25574 consensus_queue.cc:237] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:49.209622 20818 catalog_manager.cc:5484] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:49.217408 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8ac071609c4703b06265531ddc15ab" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:49.217504 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8ac071609c4703b06265531ddc15ab" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:49.217662 21225 raft_consensus.cc:2454] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:49.295572 25949 consensus_queue.cc:1035] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.297065 21208 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:49.327606 21336 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:09:49.329519 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:49.361008 21228 raft_consensus.cc:3037] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:49.362056 23710 raft_consensus.cc:1261] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:49.363299 25574 consensus_queue.cc:1035] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.363934 21228 raft_consensus.cc:1261] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:49.365202 25574 consensus_queue.cc:1035] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.365257 25949 consensus_queue.cc:1035] T 4bbbd5fb15a64e0b8495e63766c49a57 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.369552 23957 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:49.397258 20818 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 3f8ac071609c4703b06265531ddc15ab (table rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c]). Expected version 1 got 0
I20240501 20:09:49.404065 24090 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:49.405328 26035 tablet.cc:1722] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:49.409066 20818 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 3f8ac071609c4703b06265531ddc15ab (table rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c]). Expected version 1 got 0
I20240501 20:09:49.410784 26035 tablet.cc:1620] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:49.412922 23691 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:49.412606 26034 tablet.cc:1722] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:49.413796 26034 tablet.cc:1620] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:49.437260 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c] alter complete (version 1)
I20240501 20:09:49.454501 25713 consensus_queue.cc:1035] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.457499 21095 raft_consensus.cc:1261] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:09:49.458276 25840 consensus_queue.cc:1035] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:49.501875 25840 consensus_queue.cc:1035] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.505721 23710 raft_consensus.cc:1261] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:09:49.506595 25840 consensus_queue.cc:1035] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:49.526482 21076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:49.542273 20824 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet 3f8ac071609c4703b06265531ddc15ab (table rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c]). Expected version 1 got 0
I20240501 20:09:49.544126 23824 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:49.552664 20824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:49.570353 20818 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet 3f8ac071609c4703b06265531ddc15ab (table rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c]). Expected version 2 got 1
I20240501 20:09:49.578529 26035 tablet.cc:1722] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:49.579079 26035 tablet.cc:1620] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:49.584611 26034 tablet.cc:1722] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:49.585107 26034 tablet.cc:1620] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:49.594942 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c] alter complete (version 2)
I20240501 20:09:49.611788 20945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:49.620759 20818 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet 3f8ac071609c4703b06265531ddc15ab (table rebalancer_dynamic_table [id=84d87e43c4ef4901815f0b3200b73b2c]). Expected version 2 got 0
I20240501 20:09:49.611061 26050 tablet.cc:1722] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:49.623934 26050 tablet.cc:1620] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
W20240501 20:09:49.630990 24043 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 84589.2us
I20240501 20:09:49.635635 23977 raft_consensus.cc:1261] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240501 20:09:49.638432 24046 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 45920.8us
I20240501 20:09:49.640408 26046 consensus_queue.cc:1035] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 20:09:49.642372 24045 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 91192.7us
I20240501 20:09:49.646700 25946 consensus_queue.cc:1035] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.685379 25946 consensus_queue.cc:1035] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.692852 20818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:49.697466 20824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:49.702222 26050 tablet.cc:1722] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:49.702760 26050 tablet.cc:1620] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:49.711185 21076 tablet_service.cc:1467] Processing CreateTablet for tablet e1be0c0baa5e4c95915b93f659c4759f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.711699 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1be0c0baa5e4c95915b93f659c4759f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.712255 21336 tablet_service.cc:1467] Processing CreateTablet for tablet df688107929348698c8c14230c06fccc (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.711447 23691 tablet_service.cc:1467] Processing CreateTablet for tablet df688107929348698c8c14230c06fccc (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.712695 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df688107929348698c8c14230c06fccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.712846 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df688107929348698c8c14230c06fccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.714679 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 848554003579437aabd347d3c7e23e4a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.715096 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848554003579437aabd347d3c7e23e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.720093 23824 tablet_service.cc:1467] Processing CreateTablet for tablet e1be0c0baa5e4c95915b93f659c4759f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.720520 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1be0c0baa5e4c95915b93f659c4759f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.728700 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 848554003579437aabd347d3c7e23e4a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.729144 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848554003579437aabd347d3c7e23e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.729436 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 848554003579437aabd347d3c7e23e4a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.729866 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848554003579437aabd347d3c7e23e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.739723 24090 tablet_service.cc:1467] Processing CreateTablet for tablet df688107929348698c8c14230c06fccc (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.740268 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df688107929348698c8c14230c06fccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.742888 26034 tablet.cc:1722] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:49.744767 24088 tablet_service.cc:1467] Processing CreateTablet for tablet e1be0c0baa5e4c95915b93f659c4759f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.745344 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1be0c0baa5e4c95915b93f659c4759f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.761184 26035 tablet.cc:1722] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:49.765525 26062 tablet_bootstrap.cc:492] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:49.770612 26064 tablet_bootstrap.cc:492] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:49.770915 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=84d87e43c4ef4901815f0b3200b73b2c] alter complete (version 3)
I20240501 20:09:49.776911 26062 tablet_bootstrap.cc:654] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.778329 26065 tablet_bootstrap.cc:492] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:49.778827 26066 tablet_bootstrap.cc:492] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:49.778352 26064 tablet_bootstrap.cc:654] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.782716 26066 tablet_bootstrap.cc:654] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.785069 26062 tablet_bootstrap.cc:492] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:49.789882 26062 ts_tablet_manager.cc:1367] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240501 20:09:49.790082 26059 tablet_bootstrap.cc:492] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:49.792860 26065 tablet_bootstrap.cc:654] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.792752 26062 raft_consensus.cc:348] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:49.792981 26062 raft_consensus.cc:374] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.793465 26064 tablet_bootstrap.cc:492] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:49.793483 20818 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet 3f8ac071609c4703b06265531ddc15ab (table rebalancer_dynamic_table_new_name [id=84d87e43c4ef4901815f0b3200b73b2c]). Expected version 3 got 2
I20240501 20:09:49.793627 26064 ts_tablet_manager.cc:1367] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240501 20:09:49.794039 26060 tablet_bootstrap.cc:492] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:49.795624 26059 tablet_bootstrap.cc:654] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.796273 26064 raft_consensus.cc:348] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.796504 26064 raft_consensus.cc:374] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.796602 26064 raft_consensus.cc:729] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.793071 26062 raft_consensus.cc:729] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.797328 26062 consensus_queue.cc:260] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:49.798434 26062 ts_tablet_manager.cc:1398] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:49.799403 26060 tablet_bootstrap.cc:654] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.799268 26064 consensus_queue.cc:260] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.801106 26050 tablet.cc:1722] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:49.805517 26064 ts_tablet_manager.cc:1398] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240501 20:09:49.806651 26061 tablet_bootstrap.cc:492] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:49.809008 26062 tablet_bootstrap.cc:492] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:49.809998 26066 tablet_bootstrap.cc:492] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:49.811739 26064 tablet_bootstrap.cc:492] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:49.811947 26060 tablet_bootstrap.cc:492] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:49.811957 26066 ts_tablet_manager.cc:1367] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240501 20:09:49.812093 26060 ts_tablet_manager.cc:1367] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 20:09:49.812829 26061 tablet_bootstrap.cc:654] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.813083 26060 raft_consensus.cc:348] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.813303 26060 raft_consensus.cc:374] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.813385 26060 raft_consensus.cc:729] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.813689 26060 consensus_queue.cc:260] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.816610 26060 ts_tablet_manager.cc:1398] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:49.820457 26059 tablet_bootstrap.cc:492] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:49.820670 26059 ts_tablet_manager.cc:1367] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240501 20:09:49.820955 26066 raft_consensus.cc:348] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:49.821175 26066 raft_consensus.cc:374] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.821308 26066 raft_consensus.cc:729] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.821641 26062 tablet_bootstrap.cc:654] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.821816 26059 raft_consensus.cc:348] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:49.822111 26059 raft_consensus.cc:374] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.822254 26059 raft_consensus.cc:729] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.822525 26059 consensus_queue.cc:260] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:49.822827 26065 tablet_bootstrap.cc:492] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:49.822955 26064 tablet_bootstrap.cc:654] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.822983 26065 ts_tablet_manager.cc:1367] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20240501 20:09:49.823789 26065 raft_consensus.cc:348] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.823999 26065 raft_consensus.cc:374] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.824091 26065 raft_consensus.cc:729] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.824445 26059 ts_tablet_manager.cc:1398] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:49.824353 26065 consensus_queue.cc:260] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.826462 26065 ts_tablet_manager.cc:1398] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:49.828444 26066 consensus_queue.cc:260] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:49.829366 26066 ts_tablet_manager.cc:1398] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 20:09:49.840790 25785 raft_consensus.cc:482] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:49.840968 25785 raft_consensus.cc:504] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.841691 25785 leader_election.cc:288] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:49.842586 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1be0c0baa5e4c95915b93f659c4759f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:49.842622 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1be0c0baa5e4c95915b93f659c4759f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:49.842813 24108 raft_consensus.cc:2454] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:49.842832 23846 raft_consensus.cc:2454] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:49.843268 21033 leader_election.cc:302] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:49.844702 26061 tablet_bootstrap.cc:492] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:49.844877 26061 ts_tablet_manager.cc:1367] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20240501 20:09:49.845808 26061 raft_consensus.cc:348] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:49.846098 26061 raft_consensus.cc:374] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.846225 26061 raft_consensus.cc:729] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.846463 26061 consensus_queue.cc:260] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:49.847333 26061 ts_tablet_manager.cc:1398] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:49.847407 25785 raft_consensus.cc:2781] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:49.847498 25785 raft_consensus.cc:482] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:49.847632 25785 raft_consensus.cc:3037] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:49.848579 26062 tablet_bootstrap.cc:492] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:49.848753 26062 ts_tablet_manager.cc:1367] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.000s
I20240501 20:09:49.849570 26062 raft_consensus.cc:348] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:49.849756 25785 raft_consensus.cc:504] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.850041 26062 raft_consensus.cc:374] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.850162 26062 raft_consensus.cc:729] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.850420 26062 consensus_queue.cc:260] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:49.850817 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1be0c0baa5e4c95915b93f659c4759f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:49.851039 23846 raft_consensus.cc:3037] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:49.851153 26062 ts_tablet_manager.cc:1398] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:49.852430 26064 tablet_bootstrap.cc:492] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:49.852689 26064 ts_tablet_manager.cc:1367] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.006s
I20240501 20:09:49.853601 26064 raft_consensus.cc:348] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:49.853698 23846 raft_consensus.cc:2454] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:49.853901 26064 raft_consensus.cc:374] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.854038 26064 raft_consensus.cc:729] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.854362 26064 consensus_queue.cc:260] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:49.855031 26064 ts_tablet_manager.cc:1398] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:49.855326 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1be0c0baa5e4c95915b93f659c4759f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:49.855496 24108 raft_consensus.cc:3037] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:49.858124 25785 leader_election.cc:288] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:49.858247 24108 raft_consensus.cc:2454] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:49.858672 21033 leader_election.cc:302] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:49.861145 25785 raft_consensus.cc:2781] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:49.862715 25785 raft_consensus.cc:686] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:49.863013 25785 consensus_queue.cc:237] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:49.866622 20823 catalog_manager.cc:5484] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:49.878178 20823 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:49.878453 20823 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:49.881508 20823 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 3f8ac071609c4703b06265531ddc15ab
I20240501 20:09:49.882376 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8ac071609c4703b06265531ddc15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:49 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:49.882416 23689 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8ac071609c4703b06265531ddc15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:49 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:49.882671 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8ac071609c4703b06265531ddc15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:49 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:49.888039 26090 tablet_replica.cc:330] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:49.888274 26090 raft_consensus.cc:2229] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:49.888530 26090 raft_consensus.cc:2258] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:49.889016 26089 tablet_replica.cc:330] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:49.889271 26089 raft_consensus.cc:2229] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:49.889681 26089 raft_consensus.cc:2258] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:49.890259 26091 tablet_replica.cc:330] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:49.890467 26091 raft_consensus.cc:2229] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:49.890592 26089 ts_tablet_manager.cc:1875] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:49.890707 26091 raft_consensus.cc:2258] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:49.891676 26091 ts_tablet_manager.cc:1875] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:49.891819 26090 ts_tablet_manager.cc:1875] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:49.897079 26089 ts_tablet_manager.cc:1888] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:49.897218 26089 log.cc:1196] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/3f8ac071609c4703b06265531ddc15ab
I20240501 20:09:49.897646 26089 ts_tablet_manager.cc:1909] T 3f8ac071609c4703b06265531ddc15ab P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:49.898741 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 3f8ac071609c4703b06265531ddc15ab (table rebalancer_dynamic_table_new_name [id=84d87e43c4ef4901815f0b3200b73b2c]) successfully deleted
I20240501 20:09:49.898847 26091 ts_tablet_manager.cc:1888] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:49.899008 26091 log.cc:1196] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f8ac071609c4703b06265531ddc15ab
I20240501 20:09:49.899366 26091 ts_tablet_manager.cc:1909] T 3f8ac071609c4703b06265531ddc15ab P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:49.900199 26090 ts_tablet_manager.cc:1888] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:49.900365 26090 log.cc:1196] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/3f8ac071609c4703b06265531ddc15ab
I20240501 20:09:49.900535 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 3f8ac071609c4703b06265531ddc15ab (table rebalancer_dynamic_table_new_name [id=84d87e43c4ef4901815f0b3200b73b2c]) successfully deleted
I20240501 20:09:49.900859 26090 ts_tablet_manager.cc:1909] T 3f8ac071609c4703b06265531ddc15ab P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:49.901813 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 3f8ac071609c4703b06265531ddc15ab (table rebalancer_dynamic_table_new_name [id=84d87e43c4ef4901815f0b3200b73b2c]) successfully deleted
I20240501 20:09:49.985905 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:50.005704 23689 tablet_service.cc:1467] Processing CreateTablet for tablet d75991851e8d4d93a2ff6c2eecc68f09 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab5f4428dcac4095b06046db8f57ea3a]), partition=
I20240501 20:09:50.006251 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75991851e8d4d93a2ff6c2eecc68f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.010780 21076 tablet_service.cc:1467] Processing CreateTablet for tablet d75991851e8d4d93a2ff6c2eecc68f09 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab5f4428dcac4095b06046db8f57ea3a]), partition=
I20240501 20:09:50.011140 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75991851e8d4d93a2ff6c2eecc68f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.012132 23824 tablet_service.cc:1467] Processing CreateTablet for tablet d75991851e8d4d93a2ff6c2eecc68f09 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab5f4428dcac4095b06046db8f57ea3a]), partition=
I20240501 20:09:50.012460 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75991851e8d4d93a2ff6c2eecc68f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.023331 26060 tablet_bootstrap.cc:492] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:50.023979 26062 tablet_bootstrap.cc:492] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:50.025933 26060 tablet_bootstrap.cc:654] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.026005 26062 tablet_bootstrap.cc:654] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.027128 26065 tablet_bootstrap.cc:492] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:50.028621 26065 tablet_bootstrap.cc:654] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.029563 25837 raft_consensus.cc:482] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:50.029742 25837 raft_consensus.cc:504] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:50.030400 25837 leader_election.cc:288] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:50.031453 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848554003579437aabd347d3c7e23e4a" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:50.031713 23976 raft_consensus.cc:2454] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:50.032222 23644 leader_election.cc:302] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:50.032632 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848554003579437aabd347d3c7e23e4a" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:50.032892 20964 raft_consensus.cc:2454] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:50.033612 25837 raft_consensus.cc:2781] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:50.033710 25837 raft_consensus.cc:482] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:50.033820 25837 raft_consensus.cc:3037] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.038290 25837 raft_consensus.cc:504] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:50.039762 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848554003579437aabd347d3c7e23e4a" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:50.039973 23976 raft_consensus.cc:3037] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.049974 25837 leader_election.cc:288] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:50.050247 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848554003579437aabd347d3c7e23e4a" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:50.050469 20964 raft_consensus.cc:3037] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.053267 20964 raft_consensus.cc:2454] T 848554003579437aabd347d3c7e23e4a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:50.053795 23644 leader_election.cc:302] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:50.055753 23976 raft_consensus.cc:2454] T 848554003579437aabd347d3c7e23e4a P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:50.056445 25837 raft_consensus.cc:2781] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:50.056735 26065 tablet_bootstrap.cc:492] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:50.056941 25837 raft_consensus.cc:686] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:50.057041 26065 ts_tablet_manager.cc:1367] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240501 20:09:50.058384 26065 raft_consensus.cc:348] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:50.058692 26065 raft_consensus.cc:374] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.058812 26065 raft_consensus.cc:729] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.059154 26065 consensus_queue.cc:260] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:50.062323 25837 consensus_queue.cc:237] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:50.066532 26065 ts_tablet_manager.cc:1398] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:09:50.078076 20823 catalog_manager.cc:5484] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:50.092501 26062 tablet_bootstrap.cc:492] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:50.092732 26062 ts_tablet_manager.cc:1367] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.069s user 0.007s sys 0.013s
I20240501 20:09:50.093652 26062 raft_consensus.cc:348] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:50.093962 26062 raft_consensus.cc:374] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.094087 26062 raft_consensus.cc:729] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.097258 26060 tablet_bootstrap.cc:492] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:50.097477 26060 ts_tablet_manager.cc:1367] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.006s
I20240501 20:09:50.098217 26060 raft_consensus.cc:348] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:50.098440 26060 raft_consensus.cc:374] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.098515 26060 raft_consensus.cc:729] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.098778 26060 consensus_queue.cc:260] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:50.099438 26060 ts_tablet_manager.cc:1398] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:09:50.100059 26062 consensus_queue.cc:260] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:50.116719 25837 raft_consensus.cc:482] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:50.116945 25837 raft_consensus.cc:504] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:50.117333 26062 ts_tablet_manager.cc:1398] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.024s user 0.002s sys 0.001s
I20240501 20:09:50.118018 25837 leader_election.cc:288] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:50.118716 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75991851e8d4d93a2ff6c2eecc68f09" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:50.118965 23846 raft_consensus.cc:2454] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:50.119460 23647 leader_election.cc:302] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:50.119755 25837 raft_consensus.cc:2781] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:50.119874 25837 raft_consensus.cc:482] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:50.119992 25837 raft_consensus.cc:3037] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.122668 25837 raft_consensus.cc:504] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:50.124536 25837 leader_election.cc:288] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:50.124955 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75991851e8d4d93a2ff6c2eecc68f09" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:50.125169 23846 raft_consensus.cc:3037] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.128571 23846 raft_consensus.cc:2454] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:50.129056 23647 leader_election.cc:302] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:50.129542 25837 raft_consensus.cc:2781] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:50.130055 25837 raft_consensus.cc:686] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:50.130427 25837 consensus_queue.cc:237] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:50.144192 20818 catalog_manager.cc:5484] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:50.151118 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75991851e8d4d93a2ff6c2eecc68f09" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:50.151494 21095 raft_consensus.cc:2454] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:50.151919 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75991851e8d4d93a2ff6c2eecc68f09" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:50.152137 21098 raft_consensus.cc:3037] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.157142 21098 raft_consensus.cc:2454] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:50.177837 24565 raft_consensus.cc:482] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:50.178076 24565 raft_consensus.cc:504] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:50.179005 24565 leader_election.cc:288] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:50.180702 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df688107929348698c8c14230c06fccc" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:50.180711 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df688107929348698c8c14230c06fccc" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:50.180954 23710 raft_consensus.cc:2454] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:50.181034 24108 raft_consensus.cc:2454] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:50.183709 21293 leader_election.cc:302] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:50.185633 24565 raft_consensus.cc:2781] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:50.185781 24565 raft_consensus.cc:482] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:50.185900 24565 raft_consensus.cc:3037] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.193420 24565 raft_consensus.cc:504] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:50.195417 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df688107929348698c8c14230c06fccc" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:50.195719 24108 raft_consensus.cc:3037] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.199365 24108 raft_consensus.cc:2454] T df688107929348698c8c14230c06fccc P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:50.200047 21293 leader_election.cc:302] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:50.200320 24171 raft_consensus.cc:2781] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:50.200459 24171 raft_consensus.cc:686] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:50.200717 24171 consensus_queue.cc:237] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:50.202432 24565 leader_election.cc:288] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:50.202711 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df688107929348698c8c14230c06fccc" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:50.202919 23710 raft_consensus.cc:3037] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.205328 23710 raft_consensus.cc:2454] T df688107929348698c8c14230c06fccc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:50.211151 20820 catalog_manager.cc:5484] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:50.258613 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:50.267133 23846 raft_consensus.cc:1261] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:50.268168 25837 consensus_queue.cc:1035] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.269680 21098 raft_consensus.cc:1261] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:50.272907 25837 consensus_queue.cc:1035] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.282449 21336 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:09:50.284862 20818 catalog_manager.cc:5576] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271) does not have the latest schema for tablet d75991851e8d4d93a2ff6c2eecc68f09 (table rebalancer_dynamic_table [id=ab5f4428dcac4095b06046db8f57ea3a]). Expected version 1 got 0
I20240501 20:09:50.288028 26146 tablet.cc:1722] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:50.288542 26146 tablet.cc:1620] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:50.290288 26147 tablet.cc:1722] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:50.291131 26147 tablet.cc:1620] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: MemRowSet was empty: no flush needed.
I20240501 20:09:50.294735 20823 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet d75991851e8d4d93a2ff6c2eecc68f09 (table rebalancer_dynamic_table [id=ab5f4428dcac4095b06046db8f57ea3a]). Expected version 1 got 0
I20240501 20:09:50.295897 20820 catalog_manager.cc:5576] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271) does not have the latest schema for tablet d75991851e8d4d93a2ff6c2eecc68f09 (table rebalancer_dynamic_table [id=ab5f4428dcac4095b06046db8f57ea3a]). Expected version 1 got 0
I20240501 20:09:50.300083 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=ab5f4428dcac4095b06046db8f57ea3a] alter complete (version 1)
I20240501 20:09:50.320480 26148 tablet.cc:1722] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:50.321069 26148 tablet.cc:1620] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:50.323513 20818 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet d75991851e8d4d93a2ff6c2eecc68f09 (table rebalancer_dynamic_table [id=ab5f4428dcac4095b06046db8f57ea3a]). Expected version 1 got 0
I20240501 20:09:50.342742 21208 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:50.351790 24088 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:50.359977 23689 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:50.365718 23824 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:50.368223 25679 consensus_queue.cc:1035] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.385454 25785 consensus_queue.cc:1035] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.406601 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:50.408507 23957 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:50.412415 20945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:50.428680 21076 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:50.441916 26146 tablet.cc:1722] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:50.442435 26146 tablet.cc:1620] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:50.444077 26148 tablet.cc:1722] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:50.444546 26148 tablet.cc:1620] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:50.448450 26147 tablet.cc:1722] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:50.448880 26147 tablet.cc:1620] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: MemRowSet was empty: no flush needed.
I20240501 20:09:50.462000 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=ab5f4428dcac4095b06046db8f57ea3a] alter complete (version 2)
I20240501 20:09:50.515911 25837 consensus_queue.cc:1035] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.535151 25837 consensus_queue.cc:1035] T 848554003579437aabd347d3c7e23e4a P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.567366 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:50.582016 26146 tablet.cc:1722] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:50.585054 26148 tablet.cc:1722] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:50.596618 26147 tablet.cc:1722] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:50.599467 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=ab5f4428dcac4095b06046db8f57ea3a] alter complete (version 3)
I20240501 20:09:50.630400 24171 consensus_queue.cc:1035] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.643167 24171 consensus_queue.cc:1035] T df688107929348698c8c14230c06fccc P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.655668 21095 consensus_queue.cc:237] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:50.657577 23976 raft_consensus.cc:1261] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:50.658116 25785 consensus_queue.cc:1035] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:50.658615 21228 raft_consensus.cc:1261] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:50.659149 25785 consensus_queue.cc:1035] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:50.661427 25679 raft_consensus.cc:2932] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:50.662024 23976 raft_consensus.cc:2932] T a115fe66d11143ea8e24aa891e4e2f33 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:50.666117 21227 raft_consensus.cc:2932] T a115fe66d11143ea8e24aa891e4e2f33 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:50.666651 20823 catalog_manager.cc:5484] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:50.671101 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet a115fe66d11143ea8e24aa891e4e2f33 with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:50.671864 26184 tablet_replica.cc:330] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:50.672104 26184 raft_consensus.cc:2229] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:50.672420 26184 raft_consensus.cc:2258] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:50.673172 26184 ts_tablet_manager.cc:1875] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:50.679366 26184 ts_tablet_manager.cc:1888] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:09:50.679461 26184 log.cc:1196] T a115fe66d11143ea8e24aa891e4e2f33 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/a115fe66d11143ea8e24aa891e4e2f33
I20240501 20:09:50.680122 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet a115fe66d11143ea8e24aa891e4e2f33 (table rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]) successfully deleted
I20240501 20:09:50.703930 20818 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:50.704160 20818 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:50.707046 20818 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet d75991851e8d4d93a2ff6c2eecc68f09
I20240501 20:09:50.707767 23689 tablet_service.cc:1514] Processing DeleteTablet for tablet d75991851e8d4d93a2ff6c2eecc68f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:50 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:50.708156 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet d75991851e8d4d93a2ff6c2eecc68f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:50 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:50.708426 26196 tablet_replica.cc:330] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:50.708685 26196 raft_consensus.cc:2229] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:50.708704 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet d75991851e8d4d93a2ff6c2eecc68f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:50 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:50.708894 26197 tablet_replica.cc:330] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:50.709026 26196 raft_consensus.cc:2258] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:50.709154 26197 raft_consensus.cc:2229] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:50.709831 26196 ts_tablet_manager.cc:1875] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:50.709951 26197 raft_consensus.cc:2258] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:50.711504 26197 ts_tablet_manager.cc:1875] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:50.716487 26196 ts_tablet_manager.cc:1888] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:50.717077 26196 log.cc:1196] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/d75991851e8d4d93a2ff6c2eecc68f09
I20240501 20:09:50.717598 26196 ts_tablet_manager.cc:1909] T d75991851e8d4d93a2ff6c2eecc68f09 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:50.718636 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet d75991851e8d4d93a2ff6c2eecc68f09 (table rebalancer_dynamic_table_new_name [id=ab5f4428dcac4095b06046db8f57ea3a]) successfully deleted
I20240501 20:09:50.719955 26197 ts_tablet_manager.cc:1888] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:50.720158 26197 log.cc:1196] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/d75991851e8d4d93a2ff6c2eecc68f09
I20240501 20:09:50.720747 26197 ts_tablet_manager.cc:1909] T d75991851e8d4d93a2ff6c2eecc68f09 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:09:50.722128 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet d75991851e8d4d93a2ff6c2eecc68f09 (table rebalancer_dynamic_table_new_name [id=ab5f4428dcac4095b06046db8f57ea3a]) successfully deleted
I20240501 20:09:50.722580 26198 tablet_replica.cc:330] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:50.722810 26198 raft_consensus.cc:2229] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:50.723013 26198 raft_consensus.cc:2258] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:50.723722 26198 ts_tablet_manager.cc:1875] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:50.728461 26198 ts_tablet_manager.cc:1888] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:50.728585 26198 log.cc:1196] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/d75991851e8d4d93a2ff6c2eecc68f09
I20240501 20:09:50.728984 26198 ts_tablet_manager.cc:1909] T d75991851e8d4d93a2ff6c2eecc68f09 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:09:50.729876 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet d75991851e8d4d93a2ff6c2eecc68f09 (table rebalancer_dynamic_table_new_name [id=ab5f4428dcac4095b06046db8f57ea3a]) successfully deleted
I20240501 20:09:50.810763 20824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:50.825984 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 88765aa964114b7b98b57f0dc2f8abc4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=62b2e521ae864f159eab0fa1b1260b9f]), partition=
I20240501 20:09:50.826448 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88765aa964114b7b98b57f0dc2f8abc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.826762 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 88765aa964114b7b98b57f0dc2f8abc4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=62b2e521ae864f159eab0fa1b1260b9f]), partition=
I20240501 20:09:50.827224 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88765aa964114b7b98b57f0dc2f8abc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.830232 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 88765aa964114b7b98b57f0dc2f8abc4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=62b2e521ae864f159eab0fa1b1260b9f]), partition=
I20240501 20:09:50.830586 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88765aa964114b7b98b57f0dc2f8abc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.845619 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:50.854563 26225 tablet_bootstrap.cc:492] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:50.856614 26225 tablet_bootstrap.cc:654] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.859002 26225 tablet_bootstrap.cc:492] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:50.859294 26225 ts_tablet_manager.cc:1367] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:09:50.860168 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1c68b0b2184d75948c53381f5f0eb6 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:50.860672 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1c68b0b2184d75948c53381f5f0eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.860723 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1c68b0b2184d75948c53381f5f0eb6 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:50.860738 21076 tablet_service.cc:1467] Processing CreateTablet for tablet d8ef72f9e1d04bb498f540a74a850ed8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:50.861126 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ef72f9e1d04bb498f540a74a850ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.861142 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1c68b0b2184d75948c53381f5f0eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.865576 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1c68b0b2184d75948c53381f5f0eb6 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:50.866009 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1c68b0b2184d75948c53381f5f0eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.860232 26225 raft_consensus.cc:348] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:50.867347 26225 raft_consensus.cc:374] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.867448 26225 raft_consensus.cc:729] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.867565 23691 tablet_service.cc:1467] Processing CreateTablet for tablet d8ef72f9e1d04bb498f540a74a850ed8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:50.867772 26225 consensus_queue.cc:260] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:50.868006 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ef72f9e1d04bb498f540a74a850ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.868342 26225 ts_tablet_manager.cc:1398] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240501 20:09:50.871141 24090 tablet_service.cc:1467] Processing CreateTablet for tablet 726a7cd006104059b4b10e6eab407577 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:50.871402 23690 tablet_service.cc:1467] Processing CreateTablet for tablet 726a7cd006104059b4b10e6eab407577 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:50.871589 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726a7cd006104059b4b10e6eab407577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.871856 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726a7cd006104059b4b10e6eab407577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.874904 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 726a7cd006104059b4b10e6eab407577 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:50.875331 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726a7cd006104059b4b10e6eab407577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.876971 23957 tablet_service.cc:1467] Processing CreateTablet for tablet d8ef72f9e1d04bb498f540a74a850ed8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:50.877396 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ef72f9e1d04bb498f540a74a850ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:50.880966 26227 tablet_bootstrap.cc:492] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:50.881500 26228 tablet_bootstrap.cc:492] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:50.889197 26227 tablet_bootstrap.cc:654] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.890413 26228 tablet_bootstrap.cc:654] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.901162 26230 tablet_bootstrap.cc:492] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:50.904624 26230 tablet_bootstrap.cc:654] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.905959 26227 tablet_bootstrap.cc:492] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:50.906193 26227 ts_tablet_manager.cc:1367] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20240501 20:09:50.907053 26227 raft_consensus.cc:348] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:50.907346 26227 raft_consensus.cc:374] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.907476 26227 raft_consensus.cc:729] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.907814 26227 consensus_queue.cc:260] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:50.908661 26227 ts_tablet_manager.cc:1398] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:50.909097 26227 tablet_bootstrap.cc:492] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:50.911798 26227 tablet_bootstrap.cc:654] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.921239 26225 tablet_bootstrap.cc:492] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:50.924068 26225 tablet_bootstrap.cc:654] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.924422 26227 tablet_bootstrap.cc:492] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
W20240501 20:09:50.925808 24046 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 91533.5us
I20240501 20:09:50.927090 26227 ts_tablet_manager.cc:1367] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.004s
I20240501 20:09:50.928164 26227 raft_consensus.cc:348] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:50.929416 26227 raft_consensus.cc:374] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.929684 26227 raft_consensus.cc:729] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.930080 26225 tablet_bootstrap.cc:492] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:50.930174 26227 consensus_queue.cc:260] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:50.931631 26227 ts_tablet_manager.cc:1398] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:50.934528 26234 tablet_bootstrap.cc:492] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
W20240501 20:09:50.934846 21161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 59012.7us
I20240501 20:09:50.936084 26225 ts_tablet_manager.cc:1367] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20240501 20:09:50.937696 26225 raft_consensus.cc:348] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:50.940421 26225 raft_consensus.cc:374] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.940815 26225 raft_consensus.cc:729] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.941606 26225 consensus_queue.cc:260] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:50.944797 26225 ts_tablet_manager.cc:1398] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 20:09:50.945802 26234 tablet_bootstrap.cc:654] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.945829 26229 tablet_bootstrap.cc:492] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:50.945869 26230 tablet_bootstrap.cc:492] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:50.946224 26230 ts_tablet_manager.cc:1367] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.000s
I20240501 20:09:50.947652 26230 raft_consensus.cc:348] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:50.949076 26230 raft_consensus.cc:374] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.949249 26230 raft_consensus.cc:729] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.950866 26230 consensus_queue.cc:260] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:50.951819 26228 tablet_bootstrap.cc:492] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:50.952047 26228 ts_tablet_manager.cc:1367] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.013s
I20240501 20:09:50.952303 26234 tablet_bootstrap.cc:492] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:50.952486 26234 ts_tablet_manager.cc:1367] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 20:09:50.953900 26228 raft_consensus.cc:348] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:50.954214 26228 raft_consensus.cc:374] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.954277 26229 tablet_bootstrap.cc:654] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.953963 26234 raft_consensus.cc:348] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:50.954473 26234 raft_consensus.cc:374] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.954581 26234 raft_consensus.cc:729] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.954953 26234 consensus_queue.cc:260] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:50.954730 26228 raft_consensus.cc:729] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.955348 26228 consensus_queue.cc:260] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:50.956043 26228 ts_tablet_manager.cc:1398] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:50.957042 26234 ts_tablet_manager.cc:1398] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:50.957355 26234 tablet_bootstrap.cc:492] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:50.959939 26234 tablet_bootstrap.cc:654] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.956766 26228 tablet_bootstrap.cc:492] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:50.963907 26228 tablet_bootstrap.cc:654] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:50.976413 26234 tablet_bootstrap.cc:492] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:50.977567 26228 tablet_bootstrap.cc:492] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:50.977746 26228 ts_tablet_manager.cc:1367] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20240501 20:09:50.977972 26234 ts_tablet_manager.cc:1367] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240501 20:09:50.977880 26230 ts_tablet_manager.cc:1398] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:50.978526 26228 raft_consensus.cc:348] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:50.978821 26228 raft_consensus.cc:374] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.978884 26228 raft_consensus.cc:729] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.979110 26228 consensus_queue.cc:260] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:50.979656 26228 ts_tablet_manager.cc:1398] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:50.982523 21208 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:50.987687 26234 raft_consensus.cc:348] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:50.991578 26234 raft_consensus.cc:374] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:50.991727 26234 raft_consensus.cc:729] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:50.997277 26234 consensus_queue.cc:260] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:51.000149 26230 tablet_bootstrap.cc:492] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:51.000757 26234 ts_tablet_manager.cc:1398] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 20:09:51.013617 26230 tablet_bootstrap.cc:654] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.014467 21336 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:09:51.014696 25574 raft_consensus.cc:482] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:51.014859 25574 raft_consensus.cc:504] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:51.017042 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88765aa964114b7b98b57f0dc2f8abc4" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:51.017249 24110 raft_consensus.cc:2454] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:51.015514 25574 leader_election.cc:288] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:51.019831 23911 leader_election.cc:302] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:51.020186 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88765aa964114b7b98b57f0dc2f8abc4" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:51.020403 21227 raft_consensus.cc:2454] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:51.020880 25997 raft_consensus.cc:2781] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:51.020985 25997 raft_consensus.cc:482] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:51.021085 25997 raft_consensus.cc:3037] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.024717 25997 raft_consensus.cc:504] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:51.026058 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88765aa964114b7b98b57f0dc2f8abc4" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:51.026299 21227 raft_consensus.cc:3037] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.029676 21227 raft_consensus.cc:2454] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:51.030602 25997 leader_election.cc:288] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:51.031504 23912 leader_election.cc:302] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:51.032111 25997 raft_consensus.cc:2781] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:51.032469 25997 raft_consensus.cc:686] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:51.032981 25997 consensus_queue.cc:237] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:51.059700 26230 tablet_bootstrap.cc:492] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:51.059967 26230 ts_tablet_manager.cc:1367] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.000s
I20240501 20:09:51.062978 26230 raft_consensus.cc:348] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
W20240501 20:09:51.064333 23911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 43828.4us
I20240501 20:09:51.065033 26229 tablet_bootstrap.cc:492] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:51.064962 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88765aa964114b7b98b57f0dc2f8abc4" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:51.065178 26229 ts_tablet_manager.cc:1367] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.119s user 0.006s sys 0.000s
I20240501 20:09:51.065173 24108 raft_consensus.cc:3037] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.065958 26229 raft_consensus.cc:348] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.063093 26162 raft_consensus.cc:482] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:51.066370 26162 raft_consensus.cc:504] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:51.067095 26162 leader_election.cc:288] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:51.068173 24108 raft_consensus.cc:2454] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:51.072587 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c68b0b2184d75948c53381f5f0eb6" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:51.072937 23691 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:51.072947 21227 raft_consensus.cc:2454] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:51.073523 23647 leader_election.cc:302] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46; no voters:
I20240501 20:09:51.073889 26230 raft_consensus.cc:374] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.074260 26162 raft_consensus.cc:2781] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:51.075783 26162 raft_consensus.cc:482] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:51.075917 26162 raft_consensus.cc:3037] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.080754 26262 raft_consensus.cc:482] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:51.080953 26262 raft_consensus.cc:504] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.081444 26162 raft_consensus.cc:504] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:51.083943 26229 raft_consensus.cc:374] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.086027 26229 raft_consensus.cc:729] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.086382 26229 consensus_queue.cc:260] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.086783 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c68b0b2184d75948c53381f5f0eb6" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:51.087045 21227 raft_consensus.cc:3037] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.087517 26229 ts_tablet_manager.cc:1398] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240501 20:09:51.091187 21227 raft_consensus.cc:2454] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:51.091392 26162 leader_election.cc:288] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:51.092617 23647 leader_election.cc:302] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46; no voters:
I20240501 20:09:51.093003 25836 raft_consensus.cc:2781] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:51.093205 25836 raft_consensus.cc:686] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:51.093542 25836 consensus_queue.cc:237] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:51.094216 24088 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:51.098807 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef72f9e1d04bb498f540a74a850ed8" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:51.099050 21095 raft_consensus.cc:2454] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:51.101622 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef72f9e1d04bb498f540a74a850ed8" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:51.103858 23909 leader_election.cc:302] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:51.104379 26155 raft_consensus.cc:482] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:51.107290 21076 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:51.109779 26155 raft_consensus.cc:504] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:51.110499 26155 leader_election.cc:288] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:51.111037 26155 raft_consensus.cc:482] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:51.111189 26155 raft_consensus.cc:504] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:51.111776 26155 leader_election.cc:288] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:51.126324 26230 raft_consensus.cc:729] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.129632 26230 consensus_queue.cc:260] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:51.132577 20945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:51.134119 26230 ts_tablet_manager.cc:1398] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.073s user 0.002s sys 0.000s
I20240501 20:09:51.138577 26230 tablet_bootstrap.cc:492] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:51.142508 26230 tablet_bootstrap.cc:654] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.149827 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c68b0b2184d75948c53381f5f0eb6" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:51.150102 23844 raft_consensus.cc:2454] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:51.150632 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c68b0b2184d75948c53381f5f0eb6" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:51.150808 23844 raft_consensus.cc:3037] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.153707 23844 raft_consensus.cc:2454] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:51.163728 26051 raft_consensus.cc:482] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:51.163921 26051 raft_consensus.cc:504] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:51.164671 26051 leader_election.cc:288] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:51.165517 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726a7cd006104059b4b10e6eab407577" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:51.165735 23711 raft_consensus.cc:2454] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:51.166066 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726a7cd006104059b4b10e6eab407577" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:51.166213 24045 leader_election.cc:302] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:51.166344 23844 raft_consensus.cc:2454] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:51.166651 26051 raft_consensus.cc:2781] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:51.166759 26051 raft_consensus.cc:482] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:51.166832 26051 raft_consensus.cc:3037] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.169714 26051 raft_consensus.cc:504] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:51.171527 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726a7cd006104059b4b10e6eab407577" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:51.171731 26051 leader_election.cc:288] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:51.171787 23711 raft_consensus.cc:3037] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.173682 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726a7cd006104059b4b10e6eab407577" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:51.173883 23844 raft_consensus.cc:3037] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.174682 23711 raft_consensus.cc:2454] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:51.175923 24045 leader_election.cc:302] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:51.176281 26232 raft_consensus.cc:2781] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:51.176425 26232 raft_consensus.cc:686] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:51.176784 26232 consensus_queue.cc:237] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:51.176966 23844 raft_consensus.cc:2454] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:51.177430 26230 tablet_bootstrap.cc:492] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:51.177587 26230 ts_tablet_manager.cc:1367] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240501 20:09:51.177594 26257 raft_consensus.cc:2781] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:51.177821 26257 raft_consensus.cc:482] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:51.178293 26257 raft_consensus.cc:3037] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:09:51.177381 23911 leader_election.cc:341] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:51.179672 23911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 65513.8us
I20240501 20:09:51.181373 20824 catalog_manager.cc:5484] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:51.187034 20824 catalog_manager.cc:5484] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:51.187765 26262 leader_election.cc:288] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:51.186761 26230 raft_consensus.cc:348] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.189442 26230 raft_consensus.cc:374] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.189764 26230 raft_consensus.cc:729] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.193006 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c68b0b2184d75948c53381f5f0eb6" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:51.193284 21228 raft_consensus.cc:2379] T 3d1c68b0b2184d75948c53381f5f0eb6 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in this term.
I20240501 20:09:51.196576 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c68b0b2184d75948c53381f5f0eb6" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:51.196808 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726a7cd006104059b4b10e6eab407577" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:51.197062 23710 raft_consensus.cc:2379] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate 8559229e044b4e1ea661b881b3f9af50 in this term.
I20240501 20:09:51.199589 26257 raft_consensus.cc:504] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.203110 23957 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:51.203763 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726a7cd006104059b4b10e6eab407577" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:51.213259 20824 catalog_manager.cc:5484] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:51.214385 26230 consensus_queue.cc:260] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.220031 26230 ts_tablet_manager.cc:1398] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240501 20:09:51.231176 23781 leader_election.cc:302] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:09:51.231482 26267 raft_consensus.cc:2726] T 3d1c68b0b2184d75948c53381f5f0eb6 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:51.233202 23781 leader_election.cc:302] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:51.233449 26267 raft_consensus.cc:2726] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:51.240890 26257 leader_election.cc:288] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:51.240979 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef72f9e1d04bb498f540a74a850ed8" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:51.241276 21095 raft_consensus.cc:3037] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.241555 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef72f9e1d04bb498f540a74a850ed8" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:51.241772 23710 raft_consensus.cc:3037] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.245083 23710 raft_consensus.cc:2454] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:51.245695 23911 leader_election.cc:302] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:51.249625 26257 raft_consensus.cc:2781] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:51.249795 26257 raft_consensus.cc:686] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:51.250058 26257 consensus_queue.cc:237] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.252178 21095 raft_consensus.cc:2454] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:51.262521 23825 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:51.270228 20824 catalog_manager.cc:5484] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:51.296337 20824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:51.314725 24108 raft_consensus.cc:1261] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:51.315850 21228 raft_consensus.cc:1261] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:51.315896 26262 consensus_queue.cc:1035] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.316641 26262 consensus_queue.cc:1035] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.322221 20824 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 88765aa964114b7b98b57f0dc2f8abc4 (table rebalancer_dynamic_table [id=62b2e521ae864f159eab0fa1b1260b9f]). Expected version 1 got 0
I20240501 20:09:51.334704 26281 tablet.cc:1722] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:51.338447 20824 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet 88765aa964114b7b98b57f0dc2f8abc4 (table rebalancer_dynamic_table [id=62b2e521ae864f159eab0fa1b1260b9f]). Expected version 1 got 0
I20240501 20:09:51.339738 26279 tablet.cc:1722] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:51.340667 26279 tablet.cc:1620] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:51.341699 26281 tablet.cc:1620] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:51.343703 26280 tablet.cc:1722] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:51.344398 26280 tablet.cc:1620] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:51.348119 20824 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 88765aa964114b7b98b57f0dc2f8abc4 (table rebalancer_dynamic_table [id=62b2e521ae864f159eab0fa1b1260b9f]). Expected version 1 got 0
I20240501 20:09:51.355324 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=62b2e521ae864f159eab0fa1b1260b9f] alter complete (version 1)
I20240501 20:09:51.364029 21357 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756"
dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59492
I20240501 20:09:51.364277 21357 raft_consensus.cc:595] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:51.468140 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:51.474709 24565 raft_consensus.cc:979] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: : Instructing follower 4af096f5a6a9440a94f6682b399efe20 to start an election
I20240501 20:09:51.474874 24565 raft_consensus.cc:1067] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Signalling peer 4af096f5a6a9440a94f6682b399efe20 to start an election
I20240501 20:09:51.476157 21227 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756"
dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
from {username='slave'} at 127.13.251.132:55203
I20240501 20:09:51.476405 21227 raft_consensus.cc:482] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:51.476548 21227 raft_consensus.cc:3037] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:51.479327 21227 raft_consensus.cc:504] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.483824 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:51.484051 21357 raft_consensus.cc:3032] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:51.484166 21357 raft_consensus.cc:729] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:51.484447 21357 consensus_queue.cc:260] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.484946 21357 raft_consensus.cc:3037] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:51.488618 21357 raft_consensus.cc:2454] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 2.
I20240501 20:09:51.488778 26279 tablet.cc:1722] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:51.489208 21227 leader_election.cc:288] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 2 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:51.489276 26279 tablet.cc:1620] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:51.492206 26281 tablet.cc:1722] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:51.492645 26281 tablet.cc:1620] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:51.495180 26280 tablet.cc:1722] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:51.495673 26280 tablet.cc:1620] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:51.514031 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=62b2e521ae864f159eab0fa1b1260b9f] alter complete (version 2)
I20240501 20:09:51.552368 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:51.552589 20967 raft_consensus.cc:3037] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:51.555311 20967 raft_consensus.cc:2454] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 2.
I20240501 20:09:51.559082 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:51.559335 23975 raft_consensus.cc:3037] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:51.560029 26162 consensus_queue.cc:1035] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.562536 23975 raft_consensus.cc:2454] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 2.
I20240501 20:09:51.576839 21161 leader_election.cc:302] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:51.577200 25074 raft_consensus.cc:2781] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:51.577355 25074 raft_consensus.cc:686] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 2 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:51.577668 25074 consensus_queue.cc:237] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:51.588469 20818 catalog_manager.cc:5484] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 1 to 2, leader changed from 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 2 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:51.613601 26162 consensus_queue.cc:1035] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.620093 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:51.637645 26279 tablet.cc:1722] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:51.645009 26280 tablet.cc:1722] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:51.653090 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=62b2e521ae864f159eab0fa1b1260b9f] alter complete (version 3)
I20240501 20:09:51.657275 26224 consensus_queue.cc:1035] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.663931 26281 tablet.cc:1722] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:51.678038 26262 consensus_queue.cc:1035] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.702553 21208 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:51.705166 23691 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:51.719553 26232 consensus_queue.cc:1035] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.737042 25997 consensus_queue.cc:1035] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.738701 24088 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:51.760365 20820 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:51.760649 20820 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:51.764245 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 88765aa964114b7b98b57f0dc2f8abc4
I20240501 20:09:51.765215 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 88765aa964114b7b98b57f0dc2f8abc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:51 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:51.765690 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet 88765aa964114b7b98b57f0dc2f8abc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:51 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:51.765924 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 88765aa964114b7b98b57f0dc2f8abc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:51 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:51.773149 21336 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:09:51.795454 23825 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:51.801703 26355 tablet_replica.cc:330] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:51.804049 26355 raft_consensus.cc:2229] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:51.811522 26357 tablet_replica.cc:330] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:51.812160 23956 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:51.812793 26356 tablet_replica.cc:330] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:51.811839 26357 raft_consensus.cc:2229] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:51.814236 26356 raft_consensus.cc:2229] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:51.805063 26355 raft_consensus.cc:2258] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:51.817281 26355 ts_tablet_manager.cc:1875] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:51.817559 26357 raft_consensus.cc:2258] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:51.817986 26356 raft_consensus.cc:2258] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:51.820014 26357 ts_tablet_manager.cc:1875] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:51.820271 26356 ts_tablet_manager.cc:1875] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:51.837762 21076 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:51.842708 26355 ts_tablet_manager.cc:1888] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:51.842963 26355 log.cc:1196] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/88765aa964114b7b98b57f0dc2f8abc4
I20240501 20:09:51.843726 26355 ts_tablet_manager.cc:1909] T 88765aa964114b7b98b57f0dc2f8abc4 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:51.844861 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 88765aa964114b7b98b57f0dc2f8abc4 (table rebalancer_dynamic_table_new_name [id=62b2e521ae864f159eab0fa1b1260b9f]) successfully deleted
I20240501 20:09:51.857596 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:51.869702 20818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:51.869853 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 590a21c93ae543c8babdf9b89d3192e7 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.870364 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590a21c93ae543c8babdf9b89d3192e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.871675 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 590a21c93ae543c8babdf9b89d3192e7 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.872274 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590a21c93ae543c8babdf9b89d3192e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.873286 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 590a21c93ae543c8babdf9b89d3192e7 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.873692 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590a21c93ae543c8babdf9b89d3192e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.880321 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 23bf7844614647fa86b23155bf69839d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.880381 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.880780 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23bf7844614647fa86b23155bf69839d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.880832 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.885684 20945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:51.886874 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 522b77ff5e994dd0a55caeff3b27af39 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1fa41adec1b04192b8498f1f1243e21b]), partition=
I20240501 20:09:51.887250 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522b77ff5e994dd0a55caeff3b27af39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.887140 23826 tablet_service.cc:1467] Processing CreateTablet for tablet 23bf7844614647fa86b23155bf69839d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.887301 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.887526 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23bf7844614647fa86b23155bf69839d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.887867 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 23bf7844614647fa86b23155bf69839d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.888244 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23bf7844614647fa86b23155bf69839d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.892130 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.892138 23955 tablet_service.cc:1467] Processing CreateTablet for tablet 522b77ff5e994dd0a55caeff3b27af39 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1fa41adec1b04192b8498f1f1243e21b]), partition=
I20240501 20:09:51.892477 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.892611 23955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522b77ff5e994dd0a55caeff3b27af39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.896595 24090 tablet_service.cc:1467] Processing CreateTablet for tablet 522b77ff5e994dd0a55caeff3b27af39 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1fa41adec1b04192b8498f1f1243e21b]), partition=
I20240501 20:09:51.897081 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.897082 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522b77ff5e994dd0a55caeff3b27af39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.924059 26364 tablet_bootstrap.cc:492] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:51.931133 26364 tablet_bootstrap.cc:654] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.936172 26360 tablet_bootstrap.cc:492] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:51.938449 26360 tablet_bootstrap.cc:654] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.977555 26359 tablet_bootstrap.cc:492] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:51.979856 26359 tablet_bootstrap.cc:654] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.986191 26358 tablet_bootstrap.cc:492] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:51.989032 26358 tablet_bootstrap.cc:654] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.999791 26356 ts_tablet_manager.cc:1888] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:52.000730 26362 tablet_bootstrap.cc:492] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:52.001338 26363 tablet_bootstrap.cc:492] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:52.009632 26356 log.cc:1196] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/88765aa964114b7b98b57f0dc2f8abc4
I20240501 20:09:52.016402 26363 tablet_bootstrap.cc:654] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.010885 26357 ts_tablet_manager.cc:1888] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:52.026364 26356 ts_tablet_manager.cc:1909] T 88765aa964114b7b98b57f0dc2f8abc4 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:52.029848 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 88765aa964114b7b98b57f0dc2f8abc4 (table rebalancer_dynamic_table_new_name [id=62b2e521ae864f159eab0fa1b1260b9f]) successfully deleted
I20240501 20:09:52.031070 26359 tablet_bootstrap.cc:492] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:52.031343 26359 ts_tablet_manager.cc:1367] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
I20240501 20:09:52.031574 26364 tablet_bootstrap.cc:492] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:52.031806 26364 ts_tablet_manager.cc:1367] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.108s user 0.012s sys 0.013s
I20240501 20:09:52.032027 26357 log.cc:1196] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/88765aa964114b7b98b57f0dc2f8abc4
I20240501 20:09:52.032693 26359 raft_consensus.cc:348] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:52.033000 26359 raft_consensus.cc:374] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.033146 26359 raft_consensus.cc:729] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.033496 26359 consensus_queue.cc:260] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:52.034178 26364 raft_consensus.cc:348] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:52.034441 26359 ts_tablet_manager.cc:1398] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:52.034440 26364 raft_consensus.cc:374] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.034602 26364 raft_consensus.cc:729] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.034763 26357 ts_tablet_manager.cc:1909] T 88765aa964114b7b98b57f0dc2f8abc4 P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:52.034969 26364 consensus_queue.cc:260] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:52.036633 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 88765aa964114b7b98b57f0dc2f8abc4 (table rebalancer_dynamic_table_new_name [id=62b2e521ae864f159eab0fa1b1260b9f]) successfully deleted
I20240501 20:09:52.038736 26364 ts_tablet_manager.cc:1398] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 20:09:52.039330 26362 tablet_bootstrap.cc:654] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
W20240501 20:09:52.045197 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 151926us
I20240501 20:09:52.046303 26359 tablet_bootstrap.cc:492] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:52.046407 26360 tablet_bootstrap.cc:492] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:52.046660 26360 ts_tablet_manager.cc:1367] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.111s user 0.012s sys 0.012s
I20240501 20:09:52.047567 26360 raft_consensus.cc:348] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:52.047870 26360 raft_consensus.cc:374] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.048035 26360 raft_consensus.cc:729] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.048322 26363 tablet_bootstrap.cc:492] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:52.048313 26360 consensus_queue.cc:260] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:52.048525 26363 ts_tablet_manager.cc:1367] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20240501 20:09:52.049417 26363 raft_consensus.cc:348] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:52.049676 26363 raft_consensus.cc:374] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.049815 26363 raft_consensus.cc:729] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.049899 26360 ts_tablet_manager.cc:1398] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:52.050210 26360 tablet_bootstrap.cc:492] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:52.050163 26363 consensus_queue.cc:260] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:52.050892 26363 ts_tablet_manager.cc:1398] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:52.051906 26359 tablet_bootstrap.cc:654] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.053233 26360 tablet_bootstrap.cc:654] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.056907 26358 tablet_bootstrap.cc:492] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:52.057128 26358 ts_tablet_manager.cc:1367] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.008s
I20240501 20:09:52.058050 26362 tablet_bootstrap.cc:492] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:52.058007 26358 raft_consensus.cc:348] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:52.058214 26358 raft_consensus.cc:374] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.058319 26358 raft_consensus.cc:729] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.058425 26362 ts_tablet_manager.cc:1367] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.000s
I20240501 20:09:52.058538 26358 consensus_queue.cc:260] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:52.059109 26358 ts_tablet_manager.cc:1398] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:52.059458 26362 raft_consensus.cc:348] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:52.059785 26358 tablet_bootstrap.cc:492] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:52.061862 26362 raft_consensus.cc:374] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.062038 26362 raft_consensus.cc:729] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.062347 26362 consensus_queue.cc:260] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:52.068596 26362 ts_tablet_manager.cc:1398] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:09:52.069439 26358 tablet_bootstrap.cc:654] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.069845 26362 tablet_bootstrap.cc:492] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:52.073896 26362 tablet_bootstrap.cc:654] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.074652 26360 tablet_bootstrap.cc:492] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:52.075112 26360 ts_tablet_manager.cc:1367] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.004s
I20240501 20:09:52.076081 26360 raft_consensus.cc:348] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:52.076400 26360 raft_consensus.cc:374] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.076565 26360 raft_consensus.cc:729] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.076851 26360 consensus_queue.cc:260] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:52.078761 26360 ts_tablet_manager.cc:1398] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:52.080124 26358 tablet_bootstrap.cc:492] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:52.083914 26358 ts_tablet_manager.cc:1367] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.004s
I20240501 20:09:52.085012 26358 raft_consensus.cc:348] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:52.086306 26358 raft_consensus.cc:374] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.086588 26358 raft_consensus.cc:729] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.087301 26358 consensus_queue.cc:260] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:52.089951 26358 ts_tablet_manager.cc:1398] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 20:09:52.090034 26362 tablet_bootstrap.cc:492] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:52.090299 26362 ts_tablet_manager.cc:1367] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.008s
I20240501 20:09:52.091374 26358 tablet_bootstrap.cc:492] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:52.093058 26362 raft_consensus.cc:348] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:52.093362 26362 raft_consensus.cc:374] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.093472 26362 raft_consensus.cc:729] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.093765 26362 consensus_queue.cc:260] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:52.097397 26362 ts_tablet_manager.cc:1398] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 20:09:52.097790 26362 tablet_bootstrap.cc:492] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:52.100261 26358 tablet_bootstrap.cc:654] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.100891 26362 tablet_bootstrap.cc:654] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.102458 21357 raft_consensus.cc:1261] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:09:52.103483 26322 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:52.109196 26358 tablet_bootstrap.cc:492] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:52.109714 26358 ts_tablet_manager.cc:1367] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.005s
I20240501 20:09:52.110743 26358 raft_consensus.cc:348] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:52.110967 26362 tablet_bootstrap.cc:492] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:52.111135 26362 ts_tablet_manager.cc:1367] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240501 20:09:52.111287 26359 tablet_bootstrap.cc:492] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:52.111573 26359 ts_tablet_manager.cc:1367] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.010s
I20240501 20:09:52.112356 26359 raft_consensus.cc:348] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:52.113065 26359 raft_consensus.cc:374] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.113197 26359 raft_consensus.cc:729] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.113518 26359 consensus_queue.cc:260] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:52.113683 26362 raft_consensus.cc:348] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:52.113955 26362 raft_consensus.cc:374] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.114086 26362 raft_consensus.cc:729] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.114293 26359 ts_tablet_manager.cc:1398] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:52.114373 26362 consensus_queue.cc:260] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:52.115805 26358 raft_consensus.cc:374] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.115983 26358 raft_consensus.cc:729] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.116034 26362 ts_tablet_manager.cc:1398] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:52.116339 26358 consensus_queue.cc:260] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:52.118434 26358 ts_tablet_manager.cc:1398] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240501 20:09:52.128578 25997 raft_consensus.cc:482] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:52.128784 25997 raft_consensus.cc:504] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:52.129662 25997 leader_election.cc:288] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:52.130429 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522b77ff5e994dd0a55caeff3b27af39" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:52.130671 23710 raft_consensus.cc:2454] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:52.131029 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522b77ff5e994dd0a55caeff3b27af39" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:52.131256 24108 raft_consensus.cc:2454] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:52.132149 23911 leader_election.cc:302] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:52.133517 25997 raft_consensus.cc:2781] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:52.133611 25997 raft_consensus.cc:482] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:52.133702 25997 raft_consensus.cc:3037] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.137320 25997 raft_consensus.cc:504] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:52.140194 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522b77ff5e994dd0a55caeff3b27af39" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:52.140396 23710 raft_consensus.cc:3037] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.142885 23975 raft_consensus.cc:1261] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:09:52.143374 23710 raft_consensus.cc:2454] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:52.145192 26322 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:52.156215 23911 leader_election.cc:302] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:52.157433 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522b77ff5e994dd0a55caeff3b27af39" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:52.157670 24108 raft_consensus.cc:3037] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.161391 26262 raft_consensus.cc:2781] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:52.163345 24108 raft_consensus.cc:2454] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:52.163681 25997 leader_election.cc:288] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:52.183775 26268 raft_consensus.cc:482] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:52.184006 26268 raft_consensus.cc:504] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:52.185210 26268 leader_election.cc:288] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:52.186046 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224f83ad3d514a7fa5cdfa4e70ff2fd8" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:52.186173 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224f83ad3d514a7fa5cdfa4e70ff2fd8" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:52.186338 23975 raft_consensus.cc:2454] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:52.186416 24108 raft_consensus.cc:2454] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:52.186800 23781 leader_election.cc:302] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:52.187124 26268 raft_consensus.cc:2781] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:52.187271 26268 raft_consensus.cc:482] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:52.187376 26268 raft_consensus.cc:3037] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.194711 26268 raft_consensus.cc:504] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:52.195745 26268 leader_election.cc:288] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:52.196285 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224f83ad3d514a7fa5cdfa4e70ff2fd8" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:52.196509 24108 raft_consensus.cc:3037] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.197216 20967 raft_consensus.cc:1261] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:09:52.197885 26322 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:52.198872 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224f83ad3d514a7fa5cdfa4e70ff2fd8" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:52.199072 23975 raft_consensus.cc:3037] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.199276 24108 raft_consensus.cc:2454] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:52.202581 23781 leader_election.cc:302] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:52.204025 23975 raft_consensus.cc:2454] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:52.204689 26262 raft_consensus.cc:686] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:52.205071 26262 consensus_queue.cc:237] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:52.207988 26267 raft_consensus.cc:2781] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:52.231773 26267 raft_consensus.cc:686] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:52.232512 26267 consensus_queue.cc:237] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:52.244621 20820 catalog_manager.cc:5484] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
W20240501 20:09:52.271569 23779 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 47998.3us
I20240501 20:09:52.278240 20820 catalog_manager.cc:5484] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:52.340520 24565 raft_consensus.cc:482] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:52.340735 24565 raft_consensus.cc:504] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:52.341624 24565 leader_election.cc:288] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:52.342149 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590a21c93ae543c8babdf9b89d3192e7" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:52.342391 21095 raft_consensus.cc:2454] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:52.342532 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590a21c93ae543c8babdf9b89d3192e7" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:52.342769 23844 raft_consensus.cc:2454] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:52.342962 21291 leader_election.cc:302] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:52.343564 24565 raft_consensus.cc:2781] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:52.343693 24565 raft_consensus.cc:482] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:52.343799 24565 raft_consensus.cc:3037] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.345919 24565 raft_consensus.cc:504] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:52.346441 24565 leader_election.cc:288] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:52.346877 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590a21c93ae543c8babdf9b89d3192e7" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:52.346964 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590a21c93ae543c8babdf9b89d3192e7" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:52.347168 21095 raft_consensus.cc:3037] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.347174 23844 raft_consensus.cc:3037] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.349974 21095 raft_consensus.cc:2454] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:52.349963 23844 raft_consensus.cc:2454] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:52.350471 21291 leader_election.cc:302] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:52.350865 25949 raft_consensus.cc:2781] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:52.351472 25949 raft_consensus.cc:686] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:52.351984 25949 consensus_queue.cc:237] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:52.354152 26262 raft_consensus.cc:482] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:52.354353 26262 raft_consensus.cc:504] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:52.355154 26262 leader_election.cc:288] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:52.355684 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23bf7844614647fa86b23155bf69839d" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:52.355764 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23bf7844614647fa86b23155bf69839d" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:52.355906 23844 raft_consensus.cc:2454] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:52.355445 20820 catalog_manager.cc:5484] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:52.355965 21095 raft_consensus.cc:2454] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:52.356392 23909 leader_election.cc:302] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:52.356797 26262 raft_consensus.cc:2781] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:52.356906 26262 raft_consensus.cc:482] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:52.357008 26262 raft_consensus.cc:3037] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.359586 26262 raft_consensus.cc:504] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:52.360662 26262 leader_election.cc:288] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:52.360782 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23bf7844614647fa86b23155bf69839d" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:52.360929 23844 raft_consensus.cc:3037] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.362756 23844 raft_consensus.cc:2454] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:52.363283 23912 leader_election.cc:302] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:52.363297 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23bf7844614647fa86b23155bf69839d" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:52.363523 21098 raft_consensus.cc:3037] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.363709 26262 raft_consensus.cc:2781] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:52.363960 26262 raft_consensus.cc:686] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:52.364271 26262 consensus_queue.cc:237] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:52.366242 21098 raft_consensus.cc:2454] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:52.368352 20820 catalog_manager.cc:5484] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:52.391655 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:52.402420 23711 raft_consensus.cc:1261] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:52.402524 24110 raft_consensus.cc:1261] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:52.404417 26387 consensus_queue.cc:1035] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:52.405181 26387 consensus_queue.cc:1035] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:52.414126 20823 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 522b77ff5e994dd0a55caeff3b27af39 (table rebalancer_dynamic_table [id=1fa41adec1b04192b8498f1f1243e21b]). Expected version 1 got 0
I20240501 20:09:52.419335 26440 tablet.cc:1722] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:52.420419 26440 tablet.cc:1620] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:52.439301 20821 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 522b77ff5e994dd0a55caeff3b27af39 (table rebalancer_dynamic_table [id=1fa41adec1b04192b8498f1f1243e21b]). Expected version 1 got 0
I20240501 20:09:52.435981 26442 tablet.cc:1722] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:52.440544 26442 tablet.cc:1620] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:52.445997 20822 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=1fa41adec1b04192b8498f1f1243e21b] alter complete (version 1)
I20240501 20:09:52.453928 20823 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet 522b77ff5e994dd0a55caeff3b27af39 (table rebalancer_dynamic_table [id=1fa41adec1b04192b8498f1f1243e21b]). Expected version 1 got 0
I20240501 20:09:52.456383 26444 tablet.cc:1722] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:52.457108 26444 tablet.cc:1620] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:52.554944 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:52.569389 26440 tablet.cc:1722] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:52.569985 26440 tablet.cc:1620] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:52.584192 26442 tablet.cc:1722] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:52.584771 26442 tablet.cc:1620] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:52.589812 26444 tablet.cc:1722] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:52.590404 26444 tablet.cc:1620] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:52.604669 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=1fa41adec1b04192b8498f1f1243e21b] alter complete (version 2)
I20240501 20:09:52.648522 21336 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:09:52.649169 21208 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:52.653331 23691 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:52.707355 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:52.710577 23956 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:52.719079 26440 tablet.cc:1722] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:52.719828 26268 consensus_queue.cc:1035] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:52.729228 26444 tablet.cc:1722] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:52.746145 26386 consensus_queue.cc:1035] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:52.761539 26268 consensus_queue.cc:1035] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:52.791862 26262 consensus_queue.cc:1035] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:52.797220 26422 consensus_queue.cc:1035] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:52.814349 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=1fa41adec1b04192b8498f1f1243e21b] alter complete (version 3)
I20240501 20:09:52.827592 24565 consensus_queue.cc:1035] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:52.848587 26442 tablet.cc:1722] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:52.876271 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:52.881819 21076 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:52.895885 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 4341f17784bc45359a9648ab47c39628 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.896400 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4341f17784bc45359a9648ab47c39628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.896390 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 654895cdb221479da79da05bc1938876 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.896827 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654895cdb221479da79da05bc1938876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.900332 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 4341f17784bc45359a9648ab47c39628 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.900779 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4341f17784bc45359a9648ab47c39628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.904870 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 4341f17784bc45359a9648ab47c39628 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.905330 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4341f17784bc45359a9648ab47c39628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.906396 20945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:52.907985 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 654895cdb221479da79da05bc1938876 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.908408 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654895cdb221479da79da05bc1938876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.916100 20823 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:52.916314 20823 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:52.920472 20823 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 522b77ff5e994dd0a55caeff3b27af39
I20240501 20:09:52.921854 23689 tablet_service.cc:1514] Processing DeleteTablet for tablet 522b77ff5e994dd0a55caeff3b27af39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:52 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:52.922514 23955 tablet_service.cc:1514] Processing DeleteTablet for tablet 522b77ff5e994dd0a55caeff3b27af39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:52 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:52.923635 24090 tablet_service.cc:1514] Processing DeleteTablet for tablet 522b77ff5e994dd0a55caeff3b27af39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:52 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:52.925077 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 654895cdb221479da79da05bc1938876 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.925668 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654895cdb221479da79da05bc1938876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.929584 23690 tablet_service.cc:1467] Processing CreateTablet for tablet 024bb310955b4a2e80d908b8c0a8e4c1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.929631 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 024bb310955b4a2e80d908b8c0a8e4c1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.930059 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024bb310955b4a2e80d908b8c0a8e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.930090 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024bb310955b4a2e80d908b8c0a8e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.947711 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 024bb310955b4a2e80d908b8c0a8e4c1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.948316 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024bb310955b4a2e80d908b8c0a8e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.959434 24089 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:52.971485 23825 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:52.976047 26487 tablet_replica.cc:330] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:52.976444 26487 raft_consensus.cc:2229] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:52.976789 26487 raft_consensus.cc:2258] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:52.977864 26487 ts_tablet_manager.cc:1875] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:52.992967 26488 tablet_bootstrap.cc:492] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:53.001379 26484 tablet_replica.cc:330] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:53.001884 26484 raft_consensus.cc:2229] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:53.002373 26484 raft_consensus.cc:2258] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:53.004176 26481 tablet_bootstrap.cc:492] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:53.006407 26481 tablet_bootstrap.cc:654] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.016362 26483 tablet_bootstrap.cc:492] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:53.018695 26483 tablet_bootstrap.cc:654] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.023942 26486 tablet_bootstrap.cc:492] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:53.024547 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:53.028110 26486 tablet_bootstrap.cc:654] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.012203 26488 tablet_bootstrap.cc:654] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.033548 26487 ts_tablet_manager.cc:1888] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:53.033671 26487 log.cc:1196] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/522b77ff5e994dd0a55caeff3b27af39
I20240501 20:09:53.034649 26487 ts_tablet_manager.cc:1909] T 522b77ff5e994dd0a55caeff3b27af39 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:53.036231 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 522b77ff5e994dd0a55caeff3b27af39 (table rebalancer_dynamic_table_new_name [id=1fa41adec1b04192b8498f1f1243e21b]) successfully deleted
I20240501 20:09:53.039199 23957 tablet_service.cc:1467] Processing CreateTablet for tablet a2b53771a1a84add83c72361fb8574ec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=79f355151a4a429ba1925d76ed78de9d]), partition=
I20240501 20:09:53.041100 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b53771a1a84add83c72361fb8574ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.044080 26485 tablet_replica.cc:330] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:53.044339 26485 raft_consensus.cc:2229] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:53.044710 26485 raft_consensus.cc:2258] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:53.047344 26485 ts_tablet_manager.cc:1875] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:53.050323 26481 tablet_bootstrap.cc:492] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:53.050551 26481 ts_tablet_manager.cc:1367] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.004s
I20240501 20:09:53.050699 26484 ts_tablet_manager.cc:1875] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:53.051945 24088 tablet_service.cc:1467] Processing CreateTablet for tablet a2b53771a1a84add83c72361fb8574ec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=79f355151a4a429ba1925d76ed78de9d]), partition=
I20240501 20:09:53.052333 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b53771a1a84add83c72361fb8574ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.058307 20945 tablet_service.cc:1467] Processing CreateTablet for tablet a2b53771a1a84add83c72361fb8574ec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=79f355151a4a429ba1925d76ed78de9d]), partition=
I20240501 20:09:53.058756 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b53771a1a84add83c72361fb8574ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.066735 26482 tablet_bootstrap.cc:492] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:53.066797 26488 tablet_bootstrap.cc:492] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:53.066938 26488 ts_tablet_manager.cc:1367] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.000s
I20240501 20:09:53.067587 26481 raft_consensus.cc:348] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.067971 26481 raft_consensus.cc:374] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.068115 26481 raft_consensus.cc:729] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.068053 26488 raft_consensus.cc:348] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.068279 26488 raft_consensus.cc:374] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.068385 26488 raft_consensus.cc:729] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.068658 26481 consensus_queue.cc:260] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.069008 26488 consensus_queue.cc:260] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.069738 26488 ts_tablet_manager.cc:1398] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:53.070626 26481 ts_tablet_manager.cc:1398] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240501 20:09:53.070955 26481 tablet_bootstrap.cc:492] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:53.074398 26486 tablet_bootstrap.cc:492] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:53.075587 26481 tablet_bootstrap.cc:654] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.075707 26486 ts_tablet_manager.cc:1367] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.003s
I20240501 20:09:53.076967 26486 raft_consensus.cc:348] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.077492 26486 raft_consensus.cc:374] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.078802 26486 raft_consensus.cc:729] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.079241 26486 consensus_queue.cc:260] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.085709 26486 ts_tablet_manager.cc:1398] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:09:53.089269 26482 tablet_bootstrap.cc:654] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.089740 26481 tablet_bootstrap.cc:492] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:53.089896 26481 ts_tablet_manager.cc:1367] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20240501 20:09:53.090639 26493 tablet_bootstrap.cc:492] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:53.090647 26481 raft_consensus.cc:348] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.090852 26481 raft_consensus.cc:374] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.090909 26481 raft_consensus.cc:729] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.091143 26481 consensus_queue.cc:260] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.092643 26481 ts_tablet_manager.cc:1398] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 20:09:53.092983 26481 tablet_bootstrap.cc:492] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:53.094832 26481 tablet_bootstrap.cc:654] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.096568 26483 tablet_bootstrap.cc:492] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:53.097318 26488 tablet_bootstrap.cc:492] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:53.097533 26483 ts_tablet_manager.cc:1367] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.000s
I20240501 20:09:53.098485 26483 raft_consensus.cc:348] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.099205 26488 tablet_bootstrap.cc:654] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.100621 26483 raft_consensus.cc:374] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.100821 26483 raft_consensus.cc:729] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.101141 26483 consensus_queue.cc:260] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.101985 26483 ts_tablet_manager.cc:1398] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:53.110607 26481 tablet_bootstrap.cc:492] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:53.110843 26481 ts_tablet_manager.cc:1367] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240501 20:09:53.111759 26481 raft_consensus.cc:348] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.112044 26481 raft_consensus.cc:374] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.112176 26481 raft_consensus.cc:729] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.112453 26481 consensus_queue.cc:260] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.114255 26481 ts_tablet_manager.cc:1398] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:53.120205 26493 tablet_bootstrap.cc:654] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.125689 26486 tablet_bootstrap.cc:492] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:53.132745 26486 tablet_bootstrap.cc:654] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.133286 26485 ts_tablet_manager.cc:1888] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:53.133502 26485 log.cc:1196] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/522b77ff5e994dd0a55caeff3b27af39
I20240501 20:09:53.134116 26485 ts_tablet_manager.cc:1909] T 522b77ff5e994dd0a55caeff3b27af39 P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:53.134756 26482 tablet_bootstrap.cc:492] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:53.135056 26482 ts_tablet_manager.cc:1367] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20240501 20:09:53.136220 26488 tablet_bootstrap.cc:492] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:53.136241 26482 raft_consensus.cc:348] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.136440 26482 raft_consensus.cc:374] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.136536 26482 raft_consensus.cc:729] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.136561 26488 ts_tablet_manager.cc:1367] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240501 20:09:53.137482 26488 raft_consensus.cc:348] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.137789 26488 raft_consensus.cc:374] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.137912 26488 raft_consensus.cc:729] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.138325 26482 consensus_queue.cc:260] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.140048 26488 consensus_queue.cc:260] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.140751 26488 ts_tablet_manager.cc:1398] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:53.149780 26482 ts_tablet_manager.cc:1398] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 20:09:53.152515 26484 ts_tablet_manager.cc:1888] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:53.152773 26484 log.cc:1196] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/522b77ff5e994dd0a55caeff3b27af39
I20240501 20:09:53.153400 26484 ts_tablet_manager.cc:1909] T 522b77ff5e994dd0a55caeff3b27af39 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:53.156358 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 522b77ff5e994dd0a55caeff3b27af39 (table rebalancer_dynamic_table_new_name [id=1fa41adec1b04192b8498f1f1243e21b]) successfully deleted
I20240501 20:09:53.157114 26482 tablet_bootstrap.cc:492] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:53.157141 26493 tablet_bootstrap.cc:492] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:53.157285 26493 ts_tablet_manager.cc:1367] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.002s
I20240501 20:09:53.158156 26493 raft_consensus.cc:348] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.158434 26493 raft_consensus.cc:374] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.158537 26493 raft_consensus.cc:729] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.158835 26482 tablet_bootstrap.cc:654] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.158995 26493 consensus_queue.cc:260] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.160707 26493 ts_tablet_manager.cc:1398] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:53.168601 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 522b77ff5e994dd0a55caeff3b27af39 (table rebalancer_dynamic_table_new_name [id=1fa41adec1b04192b8498f1f1243e21b]) successfully deleted
I20240501 20:09:53.173974 26224 raft_consensus.cc:482] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:53.174145 26224 raft_consensus.cc:504] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.175019 26224 leader_election.cc:288] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:53.175824 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4341f17784bc45359a9648ab47c39628" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:53.176069 23975 raft_consensus.cc:2454] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:53.176961 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4341f17784bc45359a9648ab47c39628" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:53.177207 23844 raft_consensus.cc:2454] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:53.177758 24043 leader_election.cc:302] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:53.178220 26486 tablet_bootstrap.cc:492] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:53.178539 26486 ts_tablet_manager.cc:1367] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.007s
I20240501 20:09:53.178723 26224 raft_consensus.cc:2781] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:53.178823 26224 raft_consensus.cc:482] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:53.178934 26224 raft_consensus.cc:3037] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.179344 26486 raft_consensus.cc:348] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.179970 26486 raft_consensus.cc:374] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.180056 26486 raft_consensus.cc:729] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.180270 26482 tablet_bootstrap.cc:492] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:53.180317 26486 consensus_queue.cc:260] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.180438 26482 ts_tablet_manager.cc:1367] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20240501 20:09:53.182328 26482 raft_consensus.cc:348] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.182548 26482 raft_consensus.cc:374] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.182631 26482 raft_consensus.cc:729] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.182929 26482 consensus_queue.cc:260] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.183619 26482 ts_tablet_manager.cc:1398] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:53.185246 26486 ts_tablet_manager.cc:1398] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 20:09:53.190867 26482 tablet_bootstrap.cc:492] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:53.190467 26224 raft_consensus.cc:504] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.193253 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4341f17784bc45359a9648ab47c39628" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:53.193462 23975 raft_consensus.cc:3037] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.193831 26482 tablet_bootstrap.cc:654] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.196687 23975 raft_consensus.cc:2454] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:53.196997 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4341f17784bc45359a9648ab47c39628" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:53.197237 23844 raft_consensus.cc:3037] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.197204 24043 leader_election.cc:302] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:53.197552 26474 raft_consensus.cc:2781] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:53.197700 26474 raft_consensus.cc:686] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:53.198007 26474 consensus_queue.cc:237] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.199501 23844 raft_consensus.cc:2454] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:53.201151 26224 leader_election.cc:288] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:53.202591 20823 catalog_manager.cc:5484] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:53.206832 26482 tablet_bootstrap.cc:492] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:53.207072 26482 ts_tablet_manager.cc:1367] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20240501 20:09:53.209419 26482 raft_consensus.cc:348] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.209745 26482 raft_consensus.cc:374] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.210041 26482 raft_consensus.cc:729] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.210361 26482 consensus_queue.cc:260] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.212617 26482 ts_tablet_manager.cc:1398] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:53.254942 26333 raft_consensus.cc:482] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:53.255123 26333 raft_consensus.cc:504] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.257041 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024bb310955b4a2e80d908b8c0a8e4c1" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:53.257310 23844 raft_consensus.cc:2454] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:53.257719 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024bb310955b4a2e80d908b8c0a8e4c1" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:53.258023 24110 raft_consensus.cc:2454] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:53.258677 26262 raft_consensus.cc:482] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:53.258890 26262 raft_consensus.cc:504] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.259728 26262 leader_election.cc:288] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:53.267220 26333 leader_election.cc:288] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:53.267691 23646 leader_election.cc:302] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:53.267796 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654895cdb221479da79da05bc1938876" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:53.268129 23712 raft_consensus.cc:2454] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:53.268704 26507 raft_consensus.cc:2781] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:53.268853 26507 raft_consensus.cc:482] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:53.268829 23911 leader_election.cc:302] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:53.269083 26507 raft_consensus.cc:3037] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.270601 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654895cdb221479da79da05bc1938876" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:53.270823 21358 raft_consensus.cc:2454] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:53.271486 26262 raft_consensus.cc:2781] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:53.271651 26262 raft_consensus.cc:482] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:53.271777 26262 raft_consensus.cc:3037] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.272018 26507 raft_consensus.cc:504] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.272720 26507 leader_election.cc:288] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:53.274624 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024bb310955b4a2e80d908b8c0a8e4c1" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:53.274850 23844 raft_consensus.cc:3037] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.275641 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024bb310955b4a2e80d908b8c0a8e4c1" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:53.275853 24110 raft_consensus.cc:3037] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.277586 23844 raft_consensus.cc:2454] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:53.277782 24110 raft_consensus.cc:2454] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:53.273934 26262 raft_consensus.cc:504] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.278059 23647 leader_election.cc:302] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:53.278326 26507 raft_consensus.cc:2781] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:53.278587 26262 leader_election.cc:288] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:53.278983 26507 raft_consensus.cc:686] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:53.278978 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654895cdb221479da79da05bc1938876" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:53.279011 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654895cdb221479da79da05bc1938876" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:53.279204 21358 raft_consensus.cc:3037] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.279229 23712 raft_consensus.cc:3037] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.279346 26507 consensus_queue.cc:237] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:53.282042 21358 raft_consensus.cc:2454] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:53.282095 23712 raft_consensus.cc:2454] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:53.282521 23910 leader_election.cc:302] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:53.282864 26262 raft_consensus.cc:2781] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:53.283082 26262 raft_consensus.cc:686] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:53.283372 26262 consensus_queue.cc:237] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.284348 20823 catalog_manager.cc:5484] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:53.290906 20823 catalog_manager.cc:5484] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:53.300948 26375 raft_consensus.cc:482] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:53.301110 26375 raft_consensus.cc:504] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.302302 26375 leader_election.cc:288] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:53.302623 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b53771a1a84add83c72361fb8574ec" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:53.302855 23975 raft_consensus.cc:2454] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:53.302822 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b53771a1a84add83c72361fb8574ec" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:53.303059 24110 raft_consensus.cc:2454] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:53.303408 20900 leader_election.cc:302] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:53.303669 26375 raft_consensus.cc:2781] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:53.303764 26375 raft_consensus.cc:482] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:53.303844 26375 raft_consensus.cc:3037] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.305943 26375 raft_consensus.cc:504] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.306919 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b53771a1a84add83c72361fb8574ec" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:53.307077 23975 raft_consensus.cc:3037] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.307233 26375 leader_election.cc:288] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:53.307313 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b53771a1a84add83c72361fb8574ec" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:53.307509 24110 raft_consensus.cc:3037] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.309908 23975 raft_consensus.cc:2454] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:53.310298 24110 raft_consensus.cc:2454] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:53.310636 20902 leader_election.cc:302] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:53.310948 26375 raft_consensus.cc:2781] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:53.311532 26375 raft_consensus.cc:686] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:53.312090 26375 consensus_queue.cc:237] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.315289 21357 consensus_queue.cc:237] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:53.315768 20823 catalog_manager.cc:5484] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:53.317706 23975 raft_consensus.cc:1261] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:53.318329 26422 consensus_queue.cc:1035] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:53.319885 21227 raft_consensus.cc:1261] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:53.320475 25949 consensus_queue.cc:1035] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:53.321409 26422 raft_consensus.cc:2932] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:53.321879 23976 raft_consensus.cc:2932] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:53.323752 21227 raft_consensus.cc:2932] T 4587a9d5392742b58ec8d64619f2dda1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:53.332361 20823 catalog_manager.cc:5484] T 4587a9d5392742b58ec8d64619f2dda1 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:53.336326 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 4587a9d5392742b58ec8d64619f2dda1 with delete_type TABLET_DATA_TOMBSTONED (TS cda19d6a81c0473db302535ce2ada7f0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:53.336958 26530 tablet_replica.cc:330] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:53.337177 26530 raft_consensus.cc:2229] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:53.337436 26530 raft_consensus.cc:2258] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:53.338301 26530 ts_tablet_manager.cc:1875] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:53.342954 26530 ts_tablet_manager.cc:1888] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:09:53.343053 26530 log.cc:1196] T 4587a9d5392742b58ec8d64619f2dda1 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/4587a9d5392742b58ec8d64619f2dda1
I20240501 20:09:53.343684 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 4587a9d5392742b58ec8d64619f2dda1 (table rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]) successfully deleted
I20240501 20:09:53.483211 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:53.502794 24108 raft_consensus.cc:1261] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:53.503029 23976 raft_consensus.cc:1261] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:53.503650 26524 consensus_queue.cc:1035] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.504199 26524 consensus_queue.cc:1035] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.575198 20823 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet a2b53771a1a84add83c72361fb8574ec (table rebalancer_dynamic_table [id=79f355151a4a429ba1925d76ed78de9d]). Expected version 1 got 0
I20240501 20:09:53.582156 26577 tablet.cc:1722] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:53.582892 26577 tablet.cc:1620] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:53.585474 26576 tablet.cc:1722] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:53.586259 26576 tablet.cc:1620] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:53.589715 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=79f355151a4a429ba1925d76ed78de9d] alter complete (version 1)
I20240501 20:09:53.593484 20823 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet a2b53771a1a84add83c72361fb8574ec (table rebalancer_dynamic_table [id=79f355151a4a429ba1925d76ed78de9d]). Expected version 1 got 0
I20240501 20:09:53.601248 26224 consensus_queue.cc:1035] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.622277 26578 tablet.cc:1722] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:53.627269 26578 tablet.cc:1620] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:53.627684 20823 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet a2b53771a1a84add83c72361fb8574ec (table rebalancer_dynamic_table [id=79f355151a4a429ba1925d76ed78de9d]). Expected version 1 got 0
I20240501 20:09:53.640420 26224 consensus_queue.cc:1035] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.708695 26512 consensus_queue.cc:1035] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.709116 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:53.712086 26333 consensus_queue.cc:1035] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.730501 26576 tablet.cc:1722] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:53.731173 26576 tablet.cc:1620] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:53.734321 26262 consensus_queue.cc:1035] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.738509 26577 tablet.cc:1722] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:53.739040 26577 tablet.cc:1620] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:53.739727 26578 tablet.cc:1722] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:53.740263 26578 tablet.cc:1620] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:53.742393 21208 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:53.758258 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=79f355151a4a429ba1925d76ed78de9d] alter complete (version 2)
I20240501 20:09:53.776014 21336 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:09:53.796528 26333 consensus_queue.cc:1035] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.801944 23956 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:53.803462 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:53.824524 23691 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:53.850451 24088 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:53.868948 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 538f2f3892624f63bc88064e5ecd9443 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.869415 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538f2f3892624f63bc88064e5ecd9443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.871405 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 538f2f3892624f63bc88064e5ecd9443 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.871837 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538f2f3892624f63bc88064e5ecd9443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.872002 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 538f2f3892624f63bc88064e5ecd9443 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.872392 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538f2f3892624f63bc88064e5ecd9443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.869819 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:53.875860 21207 tablet_service.cc:1467] Processing CreateTablet for tablet ca4fe4379c854ff1a0b8f9f17a6311cf (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.876201 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4fe4379c854ff1a0b8f9f17a6311cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.882200 23957 tablet_service.cc:1467] Processing CreateTablet for tablet ca4fe4379c854ff1a0b8f9f17a6311cf (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.882648 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4fe4379c854ff1a0b8f9f17a6311cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.883324 23824 tablet_service.cc:1467] Processing CreateTablet for tablet ef662013a0e4494dbe319efc1e99b03b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.883808 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef662013a0e4494dbe319efc1e99b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.885609 20947 tablet_service.cc:1467] Processing CreateTablet for tablet ef662013a0e4494dbe319efc1e99b03b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.886016 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef662013a0e4494dbe319efc1e99b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.887878 20946 tablet_service.cc:1467] Processing CreateTablet for tablet ca4fe4379c854ff1a0b8f9f17a6311cf (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.888232 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4fe4379c854ff1a0b8f9f17a6311cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.882205 23956 tablet_service.cc:1467] Processing CreateTablet for tablet ef662013a0e4494dbe319efc1e99b03b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.894338 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef662013a0e4494dbe319efc1e99b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.908602 26576 tablet.cc:1722] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:53.910940 26577 tablet.cc:1722] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:53.915876 26578 tablet.cc:1722] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:53.923657 20826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=79f355151a4a429ba1925d76ed78de9d] alter complete (version 3)
I20240501 20:09:53.923826 26610 tablet_bootstrap.cc:492] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:53.926472 26610 tablet_bootstrap.cc:654] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.929876 26606 tablet_bootstrap.cc:492] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:53.933516 26606 tablet_bootstrap.cc:654] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.962067 26618 tablet_bootstrap.cc:492] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:53.968015 26618 tablet_bootstrap.cc:654] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.968644 26608 tablet_bootstrap.cc:492] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:53.970539 26608 tablet_bootstrap.cc:654] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.973008 26609 tablet_bootstrap.cc:492] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:53.976012 26609 tablet_bootstrap.cc:654] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.976922 26606 tablet_bootstrap.cc:492] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:53.977187 26606 ts_tablet_manager.cc:1367] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.007s
I20240501 20:09:53.978020 26606 raft_consensus.cc:348] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.978350 26606 raft_consensus.cc:374] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.978456 26606 raft_consensus.cc:729] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.978817 26606 consensus_queue.cc:260] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.979496 26606 ts_tablet_manager.cc:1398] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:53.981405 26606 tablet_bootstrap.cc:492] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:53.981909 20945 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:53.983062 26606 tablet_bootstrap.cc:654] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.991053 26618 tablet_bootstrap.cc:492] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:53.991286 26618 ts_tablet_manager.cc:1367] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240501 20:09:53.992121 23825 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:53.992179 26618 raft_consensus.cc:348] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:53.998871 26618 raft_consensus.cc:374] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.999037 26618 raft_consensus.cc:729] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.999620 26618 consensus_queue.cc:260] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.002741 26618 ts_tablet_manager.cc:1398] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 20:09:54.003077 26618 tablet_bootstrap.cc:492] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:54.004917 26618 tablet_bootstrap.cc:654] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.014060 26609 tablet_bootstrap.cc:492] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:54.015180 26606 tablet_bootstrap.cc:492] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:54.021096 26606 ts_tablet_manager.cc:1367] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240501 20:09:54.020880 26609 ts_tablet_manager.cc:1367] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.005s
I20240501 20:09:54.023272 26609 raft_consensus.cc:348] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.024240 26606 raft_consensus.cc:348] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:54.025199 26606 raft_consensus.cc:374] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.025480 26606 raft_consensus.cc:729] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.025919 26606 consensus_queue.cc:260] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:54.026526 26609 raft_consensus.cc:374] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.026873 26609 raft_consensus.cc:729] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.029529 26606 ts_tablet_manager.cc:1398] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 20:09:54.027443 26609 consensus_queue.cc:260] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.032160 26609 ts_tablet_manager.cc:1398] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:09:54.032444 20820 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:54.032616 20820 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:54.040084 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet a2b53771a1a84add83c72361fb8574ec
I20240501 20:09:54.040886 23956 tablet_service.cc:1514] Processing DeleteTablet for tablet a2b53771a1a84add83c72361fb8574ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:54 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:54.041494 21076 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:54.042317 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet a2b53771a1a84add83c72361fb8574ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:54 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:54.042838 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet a2b53771a1a84add83c72361fb8574ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:54 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:54.044674 26626 tablet_replica.cc:330] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:54.045697 26626 raft_consensus.cc:2229] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:54.046566 26626 raft_consensus.cc:2258] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:54.049078 26626 ts_tablet_manager.cc:1875] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:54.052652 26625 tablet_replica.cc:330] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:54.053040 26625 raft_consensus.cc:2229] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:54.056741 26608 tablet_bootstrap.cc:492] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:54.057121 26608 ts_tablet_manager.cc:1367] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.011s
I20240501 20:09:54.057361 26624 tablet_replica.cc:330] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:54.057682 26624 raft_consensus.cc:2229] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:54.058001 26624 raft_consensus.cc:2258] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:54.058048 26608 raft_consensus.cc:348] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.058408 26608 raft_consensus.cc:374] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.058550 26608 raft_consensus.cc:729] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.058974 26608 consensus_queue.cc:260] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.059767 26625 raft_consensus.cc:2258] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:54.060737 26625 ts_tablet_manager.cc:1875] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:54.061750 26608 ts_tablet_manager.cc:1398] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 20:09:54.065246 26608 tablet_bootstrap.cc:492] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:54.074043 26608 tablet_bootstrap.cc:654] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.078944 26626 ts_tablet_manager.cc:1888] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:54.079305 26626 log.cc:1196] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/a2b53771a1a84add83c72361fb8574ec
I20240501 20:09:54.079344 26625 ts_tablet_manager.cc:1888] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:54.079741 26625 log.cc:1196] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/a2b53771a1a84add83c72361fb8574ec
I20240501 20:09:54.080719 26625 ts_tablet_manager.cc:1909] T a2b53771a1a84add83c72361fb8574ec P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:54.082240 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet a2b53771a1a84add83c72361fb8574ec (table rebalancer_dynamic_table_new_name [id=79f355151a4a429ba1925d76ed78de9d]) successfully deleted
I20240501 20:09:54.082511 26610 tablet_bootstrap.cc:492] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:54.082774 26610 ts_tablet_manager.cc:1367] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.159s user 0.005s sys 0.000s
I20240501 20:09:54.082938 26626 ts_tablet_manager.cc:1909] T a2b53771a1a84add83c72361fb8574ec P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:54.083695 26610 raft_consensus.cc:348] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.084048 26610 raft_consensus.cc:374] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.084162 26610 raft_consensus.cc:729] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.084524 26610 consensus_queue.cc:260] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.086198 26610 ts_tablet_manager.cc:1398] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:54.089090 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet a2b53771a1a84add83c72361fb8574ec (table rebalancer_dynamic_table_new_name [id=79f355151a4a429ba1925d76ed78de9d]) successfully deleted
I20240501 20:09:54.093369 26608 tablet_bootstrap.cc:492] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:54.093652 26608 ts_tablet_manager.cc:1367] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20240501 20:09:54.095321 26608 raft_consensus.cc:348] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:54.095722 26608 raft_consensus.cc:374] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.095880 26608 raft_consensus.cc:729] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.096529 26608 consensus_queue.cc:260] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:54.098363 26624 ts_tablet_manager.cc:1875] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:54.099002 26608 ts_tablet_manager.cc:1398] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:54.102815 26608 tablet_bootstrap.cc:492] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:54.106122 26608 tablet_bootstrap.cc:654] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.108404 25074 raft_consensus.cc:482] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:54.108558 25074 raft_consensus.cc:504] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.109150 25074 leader_election.cc:288] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:54.110357 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538f2f3892624f63bc88064e5ecd9443" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:54.110563 20967 raft_consensus.cc:2454] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:54.111083 21161 leader_election.cc:302] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:54.111125 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538f2f3892624f63bc88064e5ecd9443" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:54.111361 23710 raft_consensus.cc:2454] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:54.111503 25074 raft_consensus.cc:2781] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:54.111594 25074 raft_consensus.cc:482] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:54.111654 25074 raft_consensus.cc:3037] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.114271 25074 raft_consensus.cc:504] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.116683 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538f2f3892624f63bc88064e5ecd9443" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:54.116891 23710 raft_consensus.cc:3037] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.119521 23710 raft_consensus.cc:2454] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:54.120087 21163 leader_election.cc:302] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46; no voters:
I20240501 20:09:54.120368 26322 raft_consensus.cc:2781] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:54.122272 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538f2f3892624f63bc88064e5ecd9443" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:54.122464 20967 raft_consensus.cc:3037] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.125334 20967 raft_consensus.cc:2454] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:54.126196 26262 raft_consensus.cc:482] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:54.126375 26262 raft_consensus.cc:504] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.127259 26262 leader_election.cc:288] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:54.127368 26608 tablet_bootstrap.cc:492] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:54.127619 26608 ts_tablet_manager.cc:1367] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.003s
I20240501 20:09:54.127910 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef662013a0e4494dbe319efc1e99b03b" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:54.128212 23844 raft_consensus.cc:2454] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:54.128662 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef662013a0e4494dbe319efc1e99b03b" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:54.128875 26608 raft_consensus.cc:348] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.129125 26608 raft_consensus.cc:374] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.129209 26608 raft_consensus.cc:729] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.129508 26608 consensus_queue.cc:260] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.129634 23912 leader_election.cc:302] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:54.129770 20964 raft_consensus.cc:2454] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:09:54.130791 26262 raft_consensus.cc:2781] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:54.130898 26262 raft_consensus.cc:482] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:54.131021 26262 raft_consensus.cc:3037] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.133911 26608 ts_tablet_manager.cc:1398] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:54.135051 26262 raft_consensus.cc:504] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.136070 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef662013a0e4494dbe319efc1e99b03b" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:54.136310 23844 raft_consensus.cc:3037] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.139108 23844 raft_consensus.cc:2454] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:54.139622 23912 leader_election.cc:302] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:54.140900 26387 raft_consensus.cc:2781] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:54.141075 26387 raft_consensus.cc:686] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:09:54.141479 26387 consensus_queue.cc:237] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.143474 25074 leader_election.cc:288] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:54.146243 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:54.149240 26262 leader_election.cc:288] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:54.149497 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef662013a0e4494dbe319efc1e99b03b" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:54.149936 20964 raft_consensus.cc:3037] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.152721 20964 raft_consensus.cc:2454] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:09:54.152801 26618 tablet_bootstrap.cc:492] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:54.152987 26618 ts_tablet_manager.cc:1367] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.150s user 0.040s sys 0.039s
I20240501 20:09:54.154882 26322 raft_consensus.cc:686] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:54.155241 26322 consensus_queue.cc:237] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.157943 23956 tablet_service.cc:1467] Processing CreateTablet for tablet bb86772f888749829873ddd617d9d948 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5222437465ba4deb8caa4d4a15906555]), partition=
I20240501 20:09:54.158111 24088 tablet_service.cc:1467] Processing CreateTablet for tablet bb86772f888749829873ddd617d9d948 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5222437465ba4deb8caa4d4a15906555]), partition=
I20240501 20:09:54.158496 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb86772f888749829873ddd617d9d948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.158509 23691 tablet_service.cc:1467] Processing CreateTablet for tablet bb86772f888749829873ddd617d9d948 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5222437465ba4deb8caa4d4a15906555]), partition=
I20240501 20:09:54.158856 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb86772f888749829873ddd617d9d948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.159616 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb86772f888749829873ddd617d9d948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.170595 26375 raft_consensus.cc:482] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:54.170750 26375 raft_consensus.cc:504] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:54.171058 26610 tablet_bootstrap.cc:492] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:54.171433 26375 leader_election.cc:288] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:54.172261 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4fe4379c854ff1a0b8f9f17a6311cf" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:54.174086 26624 ts_tablet_manager.cc:1888] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:54.174324 26610 tablet_bootstrap.cc:654] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.175858 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4fe4379c854ff1a0b8f9f17a6311cf" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:54.176065 21227 raft_consensus.cc:2454] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:54.176327 26624 log.cc:1196] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/a2b53771a1a84add83c72361fb8574ec
I20240501 20:09:54.176571 20903 leader_election.cc:302] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:54.176911 26375 raft_consensus.cc:2781] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:54.155642 26618 raft_consensus.cc:348] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:54.177017 26375 raft_consensus.cc:482] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:54.177150 26375 raft_consensus.cc:3037] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.177141 26618 raft_consensus.cc:374] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.177280 26618 raft_consensus.cc:729] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.177345 20826 catalog_manager.cc:5484] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:54.177513 26618 consensus_queue.cc:260] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:54.177789 23975 raft_consensus.cc:2454] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:54.179003 26624 ts_tablet_manager.cc:1909] T a2b53771a1a84add83c72361fb8574ec P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:54.179994 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet a2b53771a1a84add83c72361fb8574ec (table rebalancer_dynamic_table_new_name [id=79f355151a4a429ba1925d76ed78de9d]) successfully deleted
I20240501 20:09:54.180970 26375 raft_consensus.cc:504] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:54.181727 26375 leader_election.cc:288] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:54.182271 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4fe4379c854ff1a0b8f9f17a6311cf" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:54.182451 21227 raft_consensus.cc:3037] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.182478 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4fe4379c854ff1a0b8f9f17a6311cf" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:54.182649 23975 raft_consensus.cc:3037] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.185628 21227 raft_consensus.cc:2454] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:54.185984 26610 tablet_bootstrap.cc:492] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:54.186071 20903 leader_election.cc:302] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:54.186236 26610 ts_tablet_manager.cc:1367] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240501 20:09:54.186411 23975 raft_consensus.cc:2454] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:54.187206 26375 raft_consensus.cc:2781] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:54.187089 26610 raft_consensus.cc:348] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.187340 26610 raft_consensus.cc:374] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.187429 26610 raft_consensus.cc:729] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.187722 26618 ts_tablet_manager.cc:1398] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20240501 20:09:54.187701 26610 consensus_queue.cc:260] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.187985 26375 raft_consensus.cc:686] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:54.188037 26618 tablet_bootstrap.cc:492] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:54.188315 26375 consensus_queue.cc:237] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:54.189829 26618 tablet_bootstrap.cc:654] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.200426 26610 ts_tablet_manager.cc:1398] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 20:09:54.206458 26633 tablet_bootstrap.cc:492] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:54.208503 20826 catalog_manager.cc:5484] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:54.212558 20820 catalog_manager.cc:5484] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:54.228974 26633 tablet_bootstrap.cc:654] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.237277 26618 tablet_bootstrap.cc:492] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:54.237542 26618 ts_tablet_manager.cc:1367] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.008s
I20240501 20:09:54.238358 26618 raft_consensus.cc:348] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.238642 26618 raft_consensus.cc:374] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.238786 26618 raft_consensus.cc:729] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.239076 26618 consensus_queue.cc:260] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.239881 26618 ts_tablet_manager.cc:1398] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:54.256244 26633 tablet_bootstrap.cc:492] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:54.256408 26633 ts_tablet_manager.cc:1367] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240501 20:09:54.257200 26633 raft_consensus.cc:348] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.257419 26633 raft_consensus.cc:374] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.257506 26633 raft_consensus.cc:729] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.261700 26633 consensus_queue.cc:260] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.264562 26633 ts_tablet_manager.cc:1398] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:09:54.297560 26333 raft_consensus.cc:482] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:54.297758 26333 raft_consensus.cc:504] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.298605 26333 leader_election.cc:288] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:54.299122 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86772f888749829873ddd617d9d948" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:54.299405 24108 raft_consensus.cc:2454] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:54.299813 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86772f888749829873ddd617d9d948" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:54.300040 23975 raft_consensus.cc:2454] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:54.300416 23644 leader_election.cc:302] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:54.300717 26333 raft_consensus.cc:2781] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:54.300827 26333 raft_consensus.cc:482] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:54.300917 26333 raft_consensus.cc:3037] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.303778 26333 raft_consensus.cc:504] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.305012 26333 leader_election.cc:288] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:54.305455 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86772f888749829873ddd617d9d948" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:54.305546 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86772f888749829873ddd617d9d948" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:54.305665 23975 raft_consensus.cc:3037] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.305802 24108 raft_consensus.cc:3037] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.308171 23975 raft_consensus.cc:2454] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:54.308998 24108 raft_consensus.cc:2454] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:54.309446 23646 leader_election.cc:302] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:54.309824 26333 raft_consensus.cc:2781] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:54.309974 26333 raft_consensus.cc:686] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:54.310242 26333 consensus_queue.cc:237] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.321081 20826 catalog_manager.cc:5484] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:54.357789 21227 consensus_queue.cc:237] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.359874 21358 raft_consensus.cc:1261] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:54.360150 24108 raft_consensus.cc:1261] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:54.360548 26322 consensus_queue.cc:1035] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:54.360832 26630 consensus_queue.cc:1035] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:54.363288 26630 raft_consensus.cc:2932] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:54.363955 24108 raft_consensus.cc:2932] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:54.365233 21358 raft_consensus.cc:2932] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:54.369266 20826 catalog_manager.cc:5484] T 43e80df9af0e40c5acc4dea95d7e95a8 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:54.373004 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e80df9af0e40c5acc4dea95d7e95a8 with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51330
I20240501 20:09:54.384027 26659 tablet_replica.cc:330] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:09:54.384351 26659 raft_consensus.cc:2229] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:54.384617 26659 raft_consensus.cc:2258] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:54.385373 26659 ts_tablet_manager.cc:1875] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:54.391172 26659 ts_tablet_manager.cc:1888] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:09:54.391284 26659 log.cc:1196] T 43e80df9af0e40c5acc4dea95d7e95a8 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/43e80df9af0e40c5acc4dea95d7e95a8
I20240501 20:09:54.392236 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 43e80df9af0e40c5acc4dea95d7e95a8 (table rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]) successfully deleted
I20240501 20:09:54.456487 20826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:54.466423 23975 raft_consensus.cc:1261] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:54.467080 26603 consensus_queue.cc:1035] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.467739 24108 raft_consensus.cc:1261] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:54.473330 26603 consensus_queue.cc:1035] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.498374 20823 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet bb86772f888749829873ddd617d9d948 (table rebalancer_dynamic_table [id=5222437465ba4deb8caa4d4a15906555]). Expected version 1 got 0
I20240501 20:09:54.540197 26689 tablet.cc:1722] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:54.540977 26688 tablet.cc:1722] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:54.543816 26688 tablet.cc:1620] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:54.544337 26689 tablet.cc:1620] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:54.544763 20820 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet bb86772f888749829873ddd617d9d948 (table rebalancer_dynamic_table [id=5222437465ba4deb8caa4d4a15906555]). Expected version 1 got 0
I20240501 20:09:54.561766 20820 catalog_manager.cc:5576] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267) does not have the latest schema for tablet bb86772f888749829873ddd617d9d948 (table rebalancer_dynamic_table [id=5222437465ba4deb8caa4d4a15906555]). Expected version 1 got 0
I20240501 20:09:54.562057 26695 tablet.cc:1722] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:54.562860 26695 tablet.cc:1620] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:54.575533 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=5222437465ba4deb8caa4d4a15906555] alter complete (version 1)
I20240501 20:09:54.600986 26524 consensus_queue.cc:1035] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.634793 26322 consensus_queue.cc:1035] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.643213 26629 consensus_queue.cc:1035] T 538f2f3892624f63bc88064e5ecd9443 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.653443 26375 consensus_queue.cc:1035] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.682245 26387 consensus_queue.cc:1035] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.678156 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:54.694811 26688 tablet.cc:1722] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:54.695446 26688 tablet.cc:1620] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:54.696137 26689 tablet.cc:1722] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:54.711953 26387 consensus_queue.cc:1035] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240501 20:09:54.712805 26689 tablet.cc:1620] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:54.718559 26695 tablet.cc:1722] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:54.719108 26695 tablet.cc:1620] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:54.733412 20826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=5222437465ba4deb8caa4d4a15906555] alter complete (version 2)
I20240501 20:09:54.739001 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:54.748625 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 20749a5ab2744578afd2978320e7fbf3 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.749086 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20749a5ab2744578afd2978320e7fbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.749186 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 20749a5ab2744578afd2978320e7fbf3 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.749614 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20749a5ab2744578afd2978320e7fbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.749871 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 0722c64801734730abefe2a742da549e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.750262 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0722c64801734730abefe2a742da549e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.750573 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 0722c64801734730abefe2a742da549e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.750962 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0722c64801734730abefe2a742da549e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.751086 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 20749a5ab2744578afd2978320e7fbf3 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.751497 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20749a5ab2744578afd2978320e7fbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.759150 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 0722c64801734730abefe2a742da549e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.759675 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0722c64801734730abefe2a742da549e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.766644 24090 tablet_service.cc:1467] Processing CreateTablet for tablet 32a1025544cd4950ae2935425e5970a2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.766935 26724 tablet_bootstrap.cc:492] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:54.767117 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a1025544cd4950ae2935425e5970a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.769451 26724 tablet_bootstrap.cc:654] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.770489 26722 tablet_bootstrap.cc:492] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:54.775249 26722 tablet_bootstrap.cc:654] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.782156 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 32a1025544cd4950ae2935425e5970a2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.782606 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a1025544cd4950ae2935425e5970a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.784571 26728 tablet_bootstrap.cc:492] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:54.787847 26728 tablet_bootstrap.cc:654] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.789770 26724 tablet_bootstrap.cc:492] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:54.789928 26724 ts_tablet_manager.cc:1367] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.007s
I20240501 20:09:54.790427 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 32a1025544cd4950ae2935425e5970a2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.790908 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a1025544cd4950ae2935425e5970a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.791345 26724 raft_consensus.cc:348] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.791618 26724 raft_consensus.cc:374] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.791715 26724 raft_consensus.cc:729] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.792006 26724 consensus_queue.cc:260] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.797015 26724 ts_tablet_manager.cc:1398] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:09:54.798053 26733 tablet_bootstrap.cc:492] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:54.800581 26724 tablet_bootstrap.cc:492] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:54.801725 26727 tablet_bootstrap.cc:492] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:54.803290 26724 tablet_bootstrap.cc:654] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.803381 26727 tablet_bootstrap.cc:654] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.812090 26732 tablet_bootstrap.cc:492] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:54.812465 26733 tablet_bootstrap.cc:654] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.814651 26732 tablet_bootstrap.cc:654] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.817809 26724 tablet_bootstrap.cc:492] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:54.818107 26724 ts_tablet_manager.cc:1367] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20240501 20:09:54.819128 26724 raft_consensus.cc:348] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.819518 26724 raft_consensus.cc:374] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.819648 26724 raft_consensus.cc:729] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.819936 26724 consensus_queue.cc:260] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.820744 26724 ts_tablet_manager.cc:1398] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:54.821094 26724 tablet_bootstrap.cc:492] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:54.821166 26722 tablet_bootstrap.cc:492] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:54.821309 26722 ts_tablet_manager.cc:1367] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.000s
I20240501 20:09:54.822018 26722 raft_consensus.cc:348] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.823132 26724 tablet_bootstrap.cc:654] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.823962 26722 raft_consensus.cc:374] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.824193 26722 raft_consensus.cc:729] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.824576 26722 consensus_queue.cc:260] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.825301 26732 tablet_bootstrap.cc:492] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:54.825471 26732 ts_tablet_manager.cc:1367] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:09:54.826134 26732 raft_consensus.cc:348] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:54.826390 26732 raft_consensus.cc:374] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.826491 26732 raft_consensus.cc:729] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.826756 26732 consensus_queue.cc:260] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:54.827064 26722 ts_tablet_manager.cc:1398] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:54.829243 26732 ts_tablet_manager.cc:1398] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:09:54.841977 26728 tablet_bootstrap.cc:492] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:54.863847 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:54.884932 26688 tablet.cc:1722] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:54.885321 26342 raft_consensus.cc:482] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:54.885542 26342 raft_consensus.cc:504] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.886626 26342 leader_election.cc:288] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:54.887719 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20749a5ab2744578afd2978320e7fbf3" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:54.888051 23712 raft_consensus.cc:2454] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:54.888782 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20749a5ab2744578afd2978320e7fbf3" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:54.890795 26695 tablet.cc:1722] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:54.897708 21033 leader_election.cc:302] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:54.899253 26342 raft_consensus.cc:2781] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:54.899359 26342 raft_consensus.cc:482] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:54.899428 26342 raft_consensus.cc:3037] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.902196 26342 raft_consensus.cc:504] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.902830 26342 leader_election.cc:288] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:54.909487 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=5222437465ba4deb8caa4d4a15906555] alter complete (version 3)
I20240501 20:09:54.924057 26689 tablet.cc:1722] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:54.925613 26728 ts_tablet_manager.cc:1367] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.141s user 0.004s sys 0.004s
I20240501 20:09:54.926569 26728 raft_consensus.cc:348] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.926838 26728 raft_consensus.cc:374] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.926970 26728 raft_consensus.cc:729] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.927237 26728 consensus_queue.cc:260] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.929554 26728 ts_tablet_manager.cc:1398] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:54.930318 26733 tablet_bootstrap.cc:492] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:54.930507 26733 ts_tablet_manager.cc:1367] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.133s user 0.003s sys 0.000s
I20240501 20:09:54.931267 26733 raft_consensus.cc:348] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.931515 26733 raft_consensus.cc:374] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.931648 26733 raft_consensus.cc:729] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.931919 26733 consensus_queue.cc:260] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:54.938226 21208 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240501 20:09:54.946981 21033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 49310.8us
W20240501 20:09:54.947347 21033 leader_election.cc:341] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:54.948884 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20749a5ab2744578afd2978320e7fbf3" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:54.949189 23712 raft_consensus.cc:3037] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.952597 23712 raft_consensus.cc:2454] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:54.953939 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20749a5ab2744578afd2978320e7fbf3" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
W20240501 20:09:54.955394 21033 leader_election.cc:341] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:54.955734 21033 leader_election.cc:302] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters: 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:54.957598 23956 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:54.959275 26342 raft_consensus.cc:2781] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:54.959441 26342 raft_consensus.cc:686] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:54.959782 26342 consensus_queue.cc:237] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:54.974925 20826 catalog_manager.cc:5484] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:54.988317 26724 tablet_bootstrap.cc:492] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:54.988574 26724 ts_tablet_manager.cc:1367] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.168s user 0.019s sys 0.041s
I20240501 20:09:54.989570 26724 raft_consensus.cc:348] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:54.989949 26724 raft_consensus.cc:374] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.991045 26724 raft_consensus.cc:729] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.991451 26724 consensus_queue.cc:260] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:54.996959 26724 ts_tablet_manager.cc:1398] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:09:54.935524 26733 ts_tablet_manager.cc:1398] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 20:09:55.021659 20826 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:55.021867 20826 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:55.026480 20826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet bb86772f888749829873ddd617d9d948
I20240501 20:09:55.029428 23956 tablet_service.cc:1514] Processing DeleteTablet for tablet bb86772f888749829873ddd617d9d948 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:55 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:55.031282 23689 tablet_service.cc:1514] Processing DeleteTablet for tablet bb86772f888749829873ddd617d9d948 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:55 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:55.036317 23691 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:55.037428 21336 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:09:55.043033 26727 tablet_bootstrap.cc:492] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:55.043211 26727 ts_tablet_manager.cc:1367] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.242s user 0.053s sys 0.098s
I20240501 20:09:55.044039 26727 raft_consensus.cc:348] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:55.044265 26727 raft_consensus.cc:374] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:55.044358 26727 raft_consensus.cc:729] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:55.044731 26727 consensus_queue.cc:260] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:55.046470 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet bb86772f888749829873ddd617d9d948 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:55 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:55.060127 26758 tablet_replica.cc:330] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:55.060758 23824 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:55.061394 26758 raft_consensus.cc:2229] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:55.061770 26758 raft_consensus.cc:2258] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:55.062894 26758 ts_tablet_manager.cc:1875] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:55.063300 26727 ts_tablet_manager.cc:1398] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 20:09:55.063927 26762 tablet_replica.cc:330] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:55.064373 26762 raft_consensus.cc:2229] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:55.064886 26759 tablet_replica.cc:330] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:55.067958 26762 raft_consensus.cc:2258] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:55.067720 26727 tablet_bootstrap.cc:492] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:55.069370 26759 raft_consensus.cc:2229] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:55.070801 26759 raft_consensus.cc:2258] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:55.072050 26727 tablet_bootstrap.cc:654] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:55.073184 26759 ts_tablet_manager.cc:1875] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:55.072268 26762 ts_tablet_manager.cc:1875] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:55.088979 24088 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:55.094339 26762 ts_tablet_manager.cc:1888] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:55.094877 21076 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:55.095336 26727 tablet_bootstrap.cc:492] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:55.095571 26727 ts_tablet_manager.cc:1367] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240501 20:09:55.096336 26727 raft_consensus.cc:348] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:55.096554 26727 raft_consensus.cc:374] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:55.096618 26727 raft_consensus.cc:729] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:55.096858 26727 consensus_queue.cc:260] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:55.097422 26727 ts_tablet_manager.cc:1398] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:55.097553 26762 log.cc:1196] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/bb86772f888749829873ddd617d9d948
I20240501 20:09:55.098039 26762 ts_tablet_manager.cc:1909] T bb86772f888749829873ddd617d9d948 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:55.098906 26759 ts_tablet_manager.cc:1888] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:55.099115 26759 log.cc:1196] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/bb86772f888749829873ddd617d9d948
I20240501 20:09:55.100622 26759 ts_tablet_manager.cc:1909] T bb86772f888749829873ddd617d9d948 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:55.101881 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet bb86772f888749829873ddd617d9d948 (table rebalancer_dynamic_table_new_name [id=5222437465ba4deb8caa4d4a15906555]) successfully deleted
I20240501 20:09:55.111212 26758 ts_tablet_manager.cc:1888] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:55.111347 26758 log.cc:1196] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/bb86772f888749829873ddd617d9d948
I20240501 20:09:55.115456 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet bb86772f888749829873ddd617d9d948 (table rebalancer_dynamic_table_new_name [id=5222437465ba4deb8caa4d4a15906555]) successfully deleted
I20240501 20:09:55.131409 26720 raft_consensus.cc:482] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:55.131678 26720 raft_consensus.cc:504] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:55.132225 26758 ts_tablet_manager.cc:1909] T bb86772f888749829873ddd617d9d948 P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:55.132709 26720 leader_election.cc:288] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:55.134138 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0722c64801734730abefe2a742da549e" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:55.134343 20967 raft_consensus.cc:2454] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:55.134500 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet bb86772f888749829873ddd617d9d948 (table rebalancer_dynamic_table_new_name [id=5222437465ba4deb8caa4d4a15906555]) successfully deleted
I20240501 20:09:55.135089 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0722c64801734730abefe2a742da549e" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:55.135320 23712 raft_consensus.cc:2454] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:55.135985 23781 leader_election.cc:302] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:55.137095 20826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:55.138738 26476 raft_consensus.cc:2781] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:55.138834 26476 raft_consensus.cc:482] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:55.138942 26476 raft_consensus.cc:3037] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.142019 26476 raft_consensus.cc:504] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:55.146910 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0722c64801734730abefe2a742da549e" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:55.147116 23712 raft_consensus.cc:3037] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.150194 23712 raft_consensus.cc:2454] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:55.152081 23781 leader_election.cc:302] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:55.152740 26717 raft_consensus.cc:2781] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:55.152882 26717 raft_consensus.cc:686] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:55.153255 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 48ea2254c22d46b0aa2e111186ed35eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ff9130e1fd914a91a30d2aba0affe4b0]), partition=
I20240501 20:09:55.153169 26717 consensus_queue.cc:237] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:55.153648 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ea2254c22d46b0aa2e111186ed35eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.155058 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 48ea2254c22d46b0aa2e111186ed35eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ff9130e1fd914a91a30d2aba0affe4b0]), partition=
I20240501 20:09:55.155352 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ea2254c22d46b0aa2e111186ed35eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.160449 26727 tablet_bootstrap.cc:492] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:55.160789 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 48ea2254c22d46b0aa2e111186ed35eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ff9130e1fd914a91a30d2aba0affe4b0]), partition=
I20240501 20:09:55.161114 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ea2254c22d46b0aa2e111186ed35eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.162019 26727 tablet_bootstrap.cc:654] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:55.163930 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0722c64801734730abefe2a742da549e" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:55.164129 20967 raft_consensus.cc:3037] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.166841 20967 raft_consensus.cc:2454] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:55.167169 26732 tablet_bootstrap.cc:492] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:55.168463 26732 tablet_bootstrap.cc:654] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:55.167516 26476 leader_election.cc:288] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:09:55.173431 26733 tablet_bootstrap.cc:492] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:55.175199 26733 tablet_bootstrap.cc:654] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:55.183861 24565 raft_consensus.cc:482] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:55.184031 24565 raft_consensus.cc:504] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:55.184881 24565 leader_election.cc:288] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:55.188130 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a1025544cd4950ae2935425e5970a2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:55.188138 20826 catalog_manager.cc:5484] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:55.188392 24110 raft_consensus.cc:2454] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:55.188863 21293 leader_election.cc:302] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:55.192191 24565 raft_consensus.cc:2781] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:55.192284 24565 raft_consensus.cc:482] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:55.192381 24565 raft_consensus.cc:3037] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.194447 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a1025544cd4950ae2935425e5970a2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:55.194711 23712 raft_consensus.cc:2454] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:55.196343 26732 tablet_bootstrap.cc:492] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:55.196519 26732 ts_tablet_manager.cc:1367] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.005s
I20240501 20:09:55.197214 26732 raft_consensus.cc:348] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:55.198864 24565 raft_consensus.cc:504] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:55.199952 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a1025544cd4950ae2935425e5970a2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:55.200057 24565 leader_election.cc:288] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:55.200196 24110 raft_consensus.cc:3037] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.203524 24110 raft_consensus.cc:2454] T 32a1025544cd4950ae2935425e5970a2 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:55.204330 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a1025544cd4950ae2935425e5970a2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:55.204418 21293 leader_election.cc:302] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:55.204538 23710 raft_consensus.cc:3037] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.204671 24565 raft_consensus.cc:2781] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:55.204772 24565 raft_consensus.cc:686] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:55.205011 24565 consensus_queue.cc:237] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:55.206930 26732 raft_consensus.cc:374] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:55.207000 26732 raft_consensus.cc:729] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:55.207234 26732 consensus_queue.cc:260] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:55.207371 23710 raft_consensus.cc:2454] T 32a1025544cd4950ae2935425e5970a2 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:55.208266 26732 ts_tablet_manager.cc:1398] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240501 20:09:55.217517 20945 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:55.220981 26733 tablet_bootstrap.cc:492] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:55.221184 26733 ts_tablet_manager.cc:1367] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.003s
I20240501 20:09:55.222062 26733 raft_consensus.cc:348] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:55.222355 26733 raft_consensus.cc:374] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:55.222448 26733 raft_consensus.cc:729] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:55.222709 26733 consensus_queue.cc:260] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:55.223412 26733 ts_tablet_manager.cc:1398] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:55.237949 20820 catalog_manager.cc:5484] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:55.240406 26727 tablet_bootstrap.cc:492] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:55.240592 26727 ts_tablet_manager.cc:1367] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.010s
I20240501 20:09:55.244499 26727 raft_consensus.cc:348] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:55.244742 26727 raft_consensus.cc:374] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:55.244864 26727 raft_consensus.cc:729] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:55.246301 26727 consensus_queue.cc:260] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:55.248337 26727 ts_tablet_manager.cc:1398] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:09:55.271082 26474 raft_consensus.cc:482] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:55.271319 26474 raft_consensus.cc:504] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:55.272138 26474 leader_election.cc:288] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:55.273002 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ea2254c22d46b0aa2e111186ed35eb" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:55.273190 21358 raft_consensus.cc:2454] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:55.273494 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ea2254c22d46b0aa2e111186ed35eb" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:55.273761 20967 raft_consensus.cc:2454] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:55.274168 24043 leader_election.cc:302] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:55.274580 26474 raft_consensus.cc:2781] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:55.274664 26474 raft_consensus.cc:482] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:55.274772 26474 raft_consensus.cc:3037] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.279881 26474 raft_consensus.cc:504] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:55.280952 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ea2254c22d46b0aa2e111186ed35eb" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:55.280956 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ea2254c22d46b0aa2e111186ed35eb" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:55.281126 21358 raft_consensus.cc:3037] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.281126 20967 raft_consensus.cc:3037] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.283807 21358 raft_consensus.cc:2454] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:55.283836 20967 raft_consensus.cc:2454] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:55.284368 24043 leader_election.cc:302] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:55.284595 26474 leader_election.cc:288] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:55.284649 26224 raft_consensus.cc:2781] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:55.300590 26224 raft_consensus.cc:686] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:55.301079 26224 consensus_queue.cc:237] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:55.305275 20820 catalog_manager.cc:5484] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:55.361464 20967 consensus_queue.cc:237] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:55.363838 21098 raft_consensus.cc:1261] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:55.363873 23976 raft_consensus.cc:1261] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:55.364472 26524 consensus_queue.cc:1035] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:55.364951 26524 consensus_queue.cc:1035] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:55.365280 24110 raft_consensus.cc:3037] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.368057 26375 raft_consensus.cc:2932] T 68133e033c824e11a1ed744bff76a934 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:55.369186 26745 consensus_queue.cc:1035] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.370115 23976 raft_consensus.cc:2932] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:55.371752 21098 raft_consensus.cc:2932] T 68133e033c824e11a1ed744bff76a934 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:55.377274 20820 catalog_manager.cc:5484] T 68133e033c824e11a1ed744bff76a934 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:55.383991 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet 68133e033c824e11a1ed744bff76a934 with delete_type TABLET_DATA_TOMBSTONED (TS 8559229e044b4e1ea661b881b3f9af50 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:55.384734 26762 tablet_replica.cc:330] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:55.384994 26762 raft_consensus.cc:2229] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:55.385233 26762 raft_consensus.cc:2258] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:55.385885 26762 ts_tablet_manager.cc:1875] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:55.393977 26745 consensus_queue.cc:1035] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.394806 26762 ts_tablet_manager.cc:1888] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:09:55.395009 26762 log.cc:1196] T 68133e033c824e11a1ed744bff76a934 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/68133e033c824e11a1ed744bff76a934
I20240501 20:09:55.395679 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 68133e033c824e11a1ed744bff76a934 (table rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]) successfully deleted
I20240501 20:09:55.423862 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:55.429385 21358 raft_consensus.cc:1261] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:55.429632 20967 raft_consensus.cc:1261] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:55.430014 26224 consensus_queue.cc:1035] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.431023 26224 consensus_queue.cc:1035] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.438946 20820 catalog_manager.cc:5576] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399) does not have the latest schema for tablet 48ea2254c22d46b0aa2e111186ed35eb (table rebalancer_dynamic_table [id=ff9130e1fd914a91a30d2aba0affe4b0]). Expected version 1 got 0
I20240501 20:09:55.445266 26810 tablet.cc:1722] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:55.446029 26810 tablet.cc:1620] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:55.447007 26811 tablet.cc:1722] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:55.447626 26811 tablet.cc:1620] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: MemRowSet was empty: no flush needed.
I20240501 20:09:55.450103 20820 catalog_manager.cc:5576] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399) does not have the latest schema for tablet 48ea2254c22d46b0aa2e111186ed35eb (table rebalancer_dynamic_table [id=ff9130e1fd914a91a30d2aba0affe4b0]). Expected version 1 got 0
I20240501 20:09:55.464104 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=ff9130e1fd914a91a30d2aba0affe4b0] alter complete (version 1)
I20240501 20:09:55.474627 20823 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 48ea2254c22d46b0aa2e111186ed35eb (table rebalancer_dynamic_table [id=ff9130e1fd914a91a30d2aba0affe4b0]). Expected version 1 got 0
I20240501 20:09:55.492795 26817 tablet.cc:1722] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:55.493391 26817 tablet.cc:1620] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:55.493674 20823 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 48ea2254c22d46b0aa2e111186ed35eb (table rebalancer_dynamic_table [id=ff9130e1fd914a91a30d2aba0affe4b0]). Expected version 1 got 0
I20240501 20:09:55.569949 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:55.580822 26810 tablet.cc:1722] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:55.581369 26810 tablet.cc:1620] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:55.582891 26817 tablet.cc:1722] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:55.583485 26817 tablet.cc:1620] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:55.584882 26811 tablet.cc:1722] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:55.585320 26811 tablet.cc:1620] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: MemRowSet was empty: no flush needed.
I20240501 20:09:55.597193 20826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=ff9130e1fd914a91a30d2aba0affe4b0] alter complete (version 2)
I20240501 20:09:55.624897 26717 consensus_queue.cc:1035] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.650243 26717 consensus_queue.cc:1035] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.710585 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:55.745070 26810 tablet.cc:1722] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:55.746351 26811 tablet.cc:1722] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:55.759174 26817 tablet.cc:1722] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:55.762856 20826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=ff9130e1fd914a91a30d2aba0affe4b0] alter complete (version 3)
I20240501 20:09:55.793300 26422 consensus_queue.cc:1035] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.802333 23689 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:55.809854 26669 consensus_queue.cc:1035] T 32a1025544cd4950ae2935425e5970a2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.819391 21208 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:55.846200 20826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:55.850342 21336 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:09:55.858598 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffdb911cf1547329552d2e6af92c6df (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:55.859095 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffdb911cf1547329552d2e6af92c6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.866763 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffdb911cf1547329552d2e6af92c6df (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:55.867219 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffdb911cf1547329552d2e6af92c6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.869187 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffdb911cf1547329552d2e6af92c6df (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:55.869690 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffdb911cf1547329552d2e6af92c6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.873088 20819 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:55.873298 20819 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:55.877472 20819 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 48ea2254c22d46b0aa2e111186ed35eb
I20240501 20:09:55.878695 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ea2254c22d46b0aa2e111186ed35eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:55 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:55.878845 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ea2254c22d46b0aa2e111186ed35eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:55 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:55.879101 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ea2254c22d46b0aa2e111186ed35eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:55 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:55.879385 26762 tablet_replica.cc:330] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:55.879680 26762 raft_consensus.cc:2229] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:55.880129 26762 raft_consensus.cc:2258] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:55.880925 26861 tablet_replica.cc:330] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:55.881191 26762 ts_tablet_manager.cc:1875] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:55.881914 26861 raft_consensus.cc:2229] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:55.882109 26861 raft_consensus.cc:2258] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:55.884246 26861 ts_tablet_manager.cc:1875] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:55.887610 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3bdb74d3d140e397ce0b5bdef5361d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:55.888062 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3bdb74d3d140e397ce0b5bdef5361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.890343 26859 tablet_bootstrap.cc:492] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:55.892304 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3bdb74d3d140e397ce0b5bdef5361d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:55.892812 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3bdb74d3d140e397ce0b5bdef5361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.893996 26859 tablet_bootstrap.cc:654] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:55.895184 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3bdb74d3d140e397ce0b5bdef5361d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:55.896461 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3bdb74d3d140e397ce0b5bdef5361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.900799 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 2382513ca70640319674c35864e22d9c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:55.901501 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2382513ca70640319674c35864e22d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.905249 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 2382513ca70640319674c35864e22d9c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:55.905761 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2382513ca70640319674c35864e22d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.906556 26861 ts_tablet_manager.cc:1888] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:55.906674 26861 log.cc:1196] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/48ea2254c22d46b0aa2e111186ed35eb
I20240501 20:09:55.907049 26861 ts_tablet_manager.cc:1909] T 48ea2254c22d46b0aa2e111186ed35eb P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:55.907791 23955 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:55.909991 26864 tablet_bootstrap.cc:492] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:55.911732 26762 ts_tablet_manager.cc:1888] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:55.912374 26864 tablet_bootstrap.cc:654] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:55.915901 26859 tablet_bootstrap.cc:492] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:55.916030 26859 ts_tablet_manager.cc:1367] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20240501 20:09:55.916689 26859 raft_consensus.cc:348] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:55.916908 26859 raft_consensus.cc:374] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:55.916965 26859 raft_consensus.cc:729] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:55.917273 26859 consensus_queue.cc:260] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:55.922216 26859 ts_tablet_manager.cc:1398] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 20:09:55.923331 26859 tablet_bootstrap.cc:492] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:55.923631 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 48ea2254c22d46b0aa2e111186ed35eb (table rebalancer_dynamic_table_new_name [id=ff9130e1fd914a91a30d2aba0affe4b0]) successfully deleted
I20240501 20:09:55.927767 26862 tablet_replica.cc:330] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:55.928093 26862 raft_consensus.cc:2229] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:55.928413 26862 raft_consensus.cc:2258] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:55.929512 26862 ts_tablet_manager.cc:1875] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:55.929659 26859 tablet_bootstrap.cc:654] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:55.931032 26762 log.cc:1196] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/48ea2254c22d46b0aa2e111186ed35eb
I20240501 20:09:55.932291 26762 ts_tablet_manager.cc:1909] T 48ea2254c22d46b0aa2e111186ed35eb P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:55.932595 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 2382513ca70640319674c35864e22d9c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:55.933140 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2382513ca70640319674c35864e22d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:55.937038 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 48ea2254c22d46b0aa2e111186ed35eb (table rebalancer_dynamic_table_new_name [id=ff9130e1fd914a91a30d2aba0affe4b0]) successfully deleted
I20240501 20:09:55.955710 24090 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:55.956836 26872 tablet_bootstrap.cc:492] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:55.961195 26872 tablet_bootstrap.cc:654] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:55.984768 20826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:55.990760 26873 tablet_bootstrap.cc:492] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:55.992695 26873 tablet_bootstrap.cc:654] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:55.992913 21076 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:09:55.989670 26862 ts_tablet_manager.cc:1888] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:55.996861 26862 log.cc:1196] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/48ea2254c22d46b0aa2e111186ed35eb
I20240501 20:09:55.997617 26862 ts_tablet_manager.cc:1909] T 48ea2254c22d46b0aa2e111186ed35eb P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:55.999354 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 48ea2254c22d46b0aa2e111186ed35eb (table rebalancer_dynamic_table_new_name [id=ff9130e1fd914a91a30d2aba0affe4b0]) successfully deleted
I20240501 20:09:56.018643 26872 tablet_bootstrap.cc:492] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:56.019235 26872 ts_tablet_manager.cc:1367] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.000s
I20240501 20:09:56.020071 26864 tablet_bootstrap.cc:492] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:56.020319 26864 ts_tablet_manager.cc:1367] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.005s
I20240501 20:09:56.020223 26872 raft_consensus.cc:348] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.020514 26872 raft_consensus.cc:374] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.020650 26872 raft_consensus.cc:729] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.020987 26872 consensus_queue.cc:260] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.021756 23824 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:56.023290 26859 tablet_bootstrap.cc:492] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:56.027681 26859 ts_tablet_manager.cc:1367] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.104s user 0.006s sys 0.001s
I20240501 20:09:56.021390 26864 raft_consensus.cc:348] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.028643 26859 raft_consensus.cc:348] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.028928 26864 raft_consensus.cc:374] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.029072 26864 raft_consensus.cc:729] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.029392 26864 consensus_queue.cc:260] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.029873 26859 raft_consensus.cc:374] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.029992 26859 raft_consensus.cc:729] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.030328 26859 consensus_queue.cc:260] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.032068 26859 ts_tablet_manager.cc:1398] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:09:56.025259 26872 ts_tablet_manager.cc:1398] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:56.042627 26872 tablet_bootstrap.cc:492] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:56.042935 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbc34cad98b46468691ea4cdfc53e4f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89052c0e7bdc4b8e89824733ef19dd78]), partition=
I20240501 20:09:56.043411 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbc34cad98b46468691ea4cdfc53e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.045117 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbc34cad98b46468691ea4cdfc53e4f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89052c0e7bdc4b8e89824733ef19dd78]), partition=
I20240501 20:09:56.045575 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbc34cad98b46468691ea4cdfc53e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.049484 26864 ts_tablet_manager.cc:1398] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
I20240501 20:09:56.056217 26864 tablet_bootstrap.cc:492] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:56.058615 26864 tablet_bootstrap.cc:654] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.063953 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbc34cad98b46468691ea4cdfc53e4f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89052c0e7bdc4b8e89824733ef19dd78]), partition=
I20240501 20:09:56.064452 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbc34cad98b46468691ea4cdfc53e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.068809 26873 tablet_bootstrap.cc:492] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:56.069131 26873 ts_tablet_manager.cc:1367] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.006s
I20240501 20:09:56.070127 26873 raft_consensus.cc:348] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.070482 26873 raft_consensus.cc:374] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.070605 26873 raft_consensus.cc:729] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.070894 26873 consensus_queue.cc:260] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.074695 26873 ts_tablet_manager.cc:1398] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:09:56.075030 26873 tablet_bootstrap.cc:492] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:56.076628 26873 tablet_bootstrap.cc:654] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.078284 26872 tablet_bootstrap.cc:654] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.088354 26876 tablet_bootstrap.cc:492] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:56.102619 26876 tablet_bootstrap.cc:654] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.110881 26720 raft_consensus.cc:482] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:56.111104 26720 raft_consensus.cc:504] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.111851 26720 leader_election.cc:288] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:56.113790 26864 tablet_bootstrap.cc:492] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:56.114073 26864 ts_tablet_manager.cc:1367] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.001s
I20240501 20:09:56.115012 26864 raft_consensus.cc:348] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.115350 26864 raft_consensus.cc:374] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.115480 26864 raft_consensus.cc:729] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.115867 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2382513ca70640319674c35864e22d9c" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:56.115876 26864 consensus_queue.cc:260] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
W20240501 20:09:56.116799 23779 leader_election.cc:341] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:56.117159 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2382513ca70640319674c35864e22d9c" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:56.117425 24110 raft_consensus.cc:2454] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:09:56.118034 23781 leader_election.cc:302] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:56.118523 26717 raft_consensus.cc:2781] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:56.118657 26717 raft_consensus.cc:482] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:56.118762 26717 raft_consensus.cc:3037] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.120561 26864 ts_tablet_manager.cc:1398] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:09:56.121858 26717 raft_consensus.cc:504] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.124300 26745 raft_consensus.cc:482] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:56.124449 26745 raft_consensus.cc:504] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.125253 26745 leader_election.cc:288] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:56.125213 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2382513ca70640319674c35864e22d9c" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:56.125629 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3bdb74d3d140e397ce0b5bdef5361d" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:56.126034 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3bdb74d3d140e397ce0b5bdef5361d" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
W20240501 20:09:56.126166 21034 leader_election.cc:341] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:56.126266 23712 raft_consensus.cc:2454] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:56.126518 26864 tablet_bootstrap.cc:492] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:56.126713 21033 leader_election.cc:302] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters: c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:56.127072 20945 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:09:56.130349 23779 leader_election.cc:341] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:56.130627 26745 raft_consensus.cc:2781] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:56.130700 26745 raft_consensus.cc:482] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:56.130754 26745 raft_consensus.cc:3037] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.131976 26864 tablet_bootstrap.cc:654] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.145073 26876 tablet_bootstrap.cc:492] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:56.145337 26876 ts_tablet_manager.cc:1367] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.003s
I20240501 20:09:56.148243 26717 leader_election.cc:288] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:56.149633 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2382513ca70640319674c35864e22d9c" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:56.149852 24110 raft_consensus.cc:3037] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.153889 26876 raft_consensus.cc:348] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.154953 26876 raft_consensus.cc:374] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.155125 26876 raft_consensus.cc:729] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.155256 26745 raft_consensus.cc:504] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.155493 26876 consensus_queue.cc:260] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.156603 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3bdb74d3d140e397ce0b5bdef5361d" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:56.156826 23712 raft_consensus.cc:3037] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.158198 26876 ts_tablet_manager.cc:1398] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240501 20:09:56.158546 26876 tablet_bootstrap.cc:492] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:56.159282 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3bdb74d3d140e397ce0b5bdef5361d" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
W20240501 20:09:56.159849 21034 leader_election.cc:341] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:56.160095 26745 leader_election.cc:288] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:56.159503 23712 raft_consensus.cc:2454] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:56.160629 21033 leader_election.cc:302] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters: c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:56.161087 26745 raft_consensus.cc:2781] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:56.161316 26745 raft_consensus.cc:686] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:56.161636 26745 consensus_queue.cc:237] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.162474 26876 tablet_bootstrap.cc:654] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.162866 24110 raft_consensus.cc:2454] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:09:56.163375 23781 leader_election.cc:302] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:56.164151 26720 raft_consensus.cc:2781] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:56.164300 26720 raft_consensus.cc:686] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:56.164606 26720 consensus_queue.cc:237] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.184991 26876 tablet_bootstrap.cc:492] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:56.185279 26876 ts_tablet_manager.cc:1367] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20240501 20:09:56.186286 26876 raft_consensus.cc:348] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.186635 26876 raft_consensus.cc:374] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.186877 26876 raft_consensus.cc:729] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.187227 26876 consensus_queue.cc:260] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.189987 26876 ts_tablet_manager.cc:1398] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:09:56.202315 20826 catalog_manager.cc:5484] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:56.206369 20822 catalog_manager.cc:5484] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:56.209264 26872 tablet_bootstrap.cc:492] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:56.209513 26872 ts_tablet_manager.cc:1367] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.167s user 0.013s sys 0.003s
I20240501 20:09:56.210667 26872 raft_consensus.cc:348] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.211001 26872 raft_consensus.cc:374] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.211205 26872 raft_consensus.cc:729] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.211527 26872 consensus_queue.cc:260] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.212165 26872 ts_tablet_manager.cc:1398] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:56.213769 26872 tablet_bootstrap.cc:492] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:56.217413 26872 tablet_bootstrap.cc:654] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.218314 26745 raft_consensus.cc:482] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:56.218509 26745 raft_consensus.cc:504] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.219682 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffdb911cf1547329552d2e6af92c6df" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:56.219942 23712 raft_consensus.cc:2454] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:56.221509 21033 leader_election.cc:302] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:56.229411 26749 raft_consensus.cc:2781] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:56.229514 26749 raft_consensus.cc:482] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:56.229605 26749 raft_consensus.cc:3037] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.230016 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffdb911cf1547329552d2e6af92c6df" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:56.230268 23975 raft_consensus.cc:2454] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:09:56.231313 26745 leader_election.cc:288] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:56.232189 26749 raft_consensus.cc:504] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.232673 26749 leader_election.cc:288] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:56.233155 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffdb911cf1547329552d2e6af92c6df" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:56.233368 23975 raft_consensus.cc:3037] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.234748 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffdb911cf1547329552d2e6af92c6df" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:56.234975 23710 raft_consensus.cc:3037] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.236048 26512 raft_consensus.cc:482] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:56.236274 26864 tablet_bootstrap.cc:492] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:56.236202 26512 raft_consensus.cc:504] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.236435 26864 ts_tablet_manager.cc:1367] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.110s user 0.007s sys 0.002s
I20240501 20:09:56.236855 26512 leader_election.cc:288] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:56.237540 26864 raft_consensus.cc:348] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:56.237746 26864 raft_consensus.cc:374] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.237891 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2382513ca70640319674c35864e22d9c" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:56.237869 23710 raft_consensus.cc:2454] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:56.238514 23975 raft_consensus.cc:2454] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:09:56.238447 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2382513ca70640319674c35864e22d9c" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:56.238718 24110 raft_consensus.cc:2379] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 744c5a532e724391830c0785d32eb2fb in current term 1: Already voted for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in this term.
I20240501 20:09:56.238881 21031 leader_election.cc:302] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:56.239640 26749 raft_consensus.cc:2781] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:56.239647 23912 leader_election.cc:302] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb; no voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:56.239758 26749 raft_consensus.cc:686] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:09:56.239988 26749 consensus_queue.cc:237] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.240893 26512 raft_consensus.cc:3037] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.237860 26864 raft_consensus.cc:729] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.243216 26864 consensus_queue.cc:260] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:56.243850 26512 raft_consensus.cc:2726] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:56.259021 26864 ts_tablet_manager.cc:1398] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240501 20:09:56.258569 26872 tablet_bootstrap.cc:492] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:56.263305 26872 ts_tablet_manager.cc:1367] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.001s
I20240501 20:09:56.264170 26872 raft_consensus.cc:348] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:56.264425 26872 raft_consensus.cc:374] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.264096 20822 catalog_manager.cc:5484] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:56.264533 26872 raft_consensus.cc:729] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.264750 26872 consensus_queue.cc:260] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:56.265947 26872 ts_tablet_manager.cc:1398] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:56.268626 26873 tablet_bootstrap.cc:492] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:56.268855 26873 ts_tablet_manager.cc:1367] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.194s user 0.007s sys 0.002s
I20240501 20:09:56.269760 26873 raft_consensus.cc:348] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:56.270123 26873 raft_consensus.cc:374] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.270251 26873 raft_consensus.cc:729] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.273032 26873 consensus_queue.cc:260] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:56.284325 26873 ts_tablet_manager.cc:1398] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 20:09:56.318069 21358 consensus_queue.cc:237] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:56.320708 23710 raft_consensus.cc:1261] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:56.320693 20964 raft_consensus.cc:1261] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:56.321317 24565 consensus_queue.cc:1035] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:56.321575 26669 consensus_queue.cc:1035] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:56.324553 24565 raft_consensus.cc:2932] T aebb2a23076449fd9db492ed3e3b0cbf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:56.325632 26720 raft_consensus.cc:482] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:56.325803 26720 raft_consensus.cc:504] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:56.325773 23710 raft_consensus.cc:2932] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:56.326498 26720 leader_election.cc:288] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:56.326972 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3bdb74d3d140e397ce0b5bdef5361d" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:56.327243 23712 raft_consensus.cc:2379] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate feac73db0a3b4aafa374ca5dfd322b08 in this term.
I20240501 20:09:56.327800 20964 raft_consensus.cc:2932] T aebb2a23076449fd9db492ed3e3b0cbf P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:56.331808 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3bdb74d3d140e397ce0b5bdef5361d" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:56.332502 23779 leader_election.cc:302] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:09:56.332677 20819 catalog_manager.cc:5484] T aebb2a23076449fd9db492ed3e3b0cbf P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:56.340675 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet aebb2a23076449fd9db492ed3e3b0cbf with delete_type TABLET_DATA_TOMBSTONED (TS 4af096f5a6a9440a94f6682b399efe20 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:56.341254 26720 raft_consensus.cc:3037] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.342300 26912 tablet_replica.cc:330] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:56.342518 26912 raft_consensus.cc:2229] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:56.342836 26912 raft_consensus.cc:2258] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:56.343672 26912 ts_tablet_manager.cc:1875] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:56.343786 26720 raft_consensus.cc:2726] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:56.345254 26474 raft_consensus.cc:482] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:56.345459 26474 raft_consensus.cc:504] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:56.346434 26474 leader_election.cc:288] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:56.347630 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbc34cad98b46468691ea4cdfc53e4f" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:56.347854 23710 raft_consensus.cc:2454] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:56.348230 24045 leader_election.cc:302] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:56.348616 26474 raft_consensus.cc:2781] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:56.348702 26474 raft_consensus.cc:482] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:56.348793 26474 raft_consensus.cc:3037] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.348942 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbc34cad98b46468691ea4cdfc53e4f" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:56.349159 23844 raft_consensus.cc:2454] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:56.350868 26912 ts_tablet_manager.cc:1888] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:09:56.351095 26912 log.cc:1196] T aebb2a23076449fd9db492ed3e3b0cbf P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/aebb2a23076449fd9db492ed3e3b0cbf
I20240501 20:09:56.352165 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet aebb2a23076449fd9db492ed3e3b0cbf (table rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]) successfully deleted
I20240501 20:09:56.352696 26474 raft_consensus.cc:504] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:56.354760 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbc34cad98b46468691ea4cdfc53e4f" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:09:56.355008 23710 raft_consensus.cc:3037] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.356060 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbc34cad98b46468691ea4cdfc53e4f" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:56.356302 23844 raft_consensus.cc:3037] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:56.358359 23710 raft_consensus.cc:2454] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:56.359891 24045 leader_election.cc:302] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:56.360298 26224 raft_consensus.cc:2781] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:56.360431 26224 raft_consensus.cc:686] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:56.360707 26224 consensus_queue.cc:237] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:09:56.361069 23844 raft_consensus.cc:2454] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:56.362440 26474 leader_election.cc:288] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:56.366500 20819 catalog_manager.cc:5484] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:09:56.496713 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:56.513450 23844 raft_consensus.cc:1261] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:56.514525 23710 raft_consensus.cc:1261] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:56.515255 26474 consensus_queue.cc:1035] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:56.515645 26769 consensus_queue.cc:1035] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:56.569041 20818 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 7bbc34cad98b46468691ea4cdfc53e4f (table rebalancer_dynamic_table [id=89052c0e7bdc4b8e89824733ef19dd78]). Expected version 1 got 0
I20240501 20:09:56.571771 26953 tablet.cc:1722] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:56.572528 26953 tablet.cc:1620] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:56.576864 26745 consensus_queue.cc:1035] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:56.589869 20826 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 7bbc34cad98b46468691ea4cdfc53e4f (table rebalancer_dynamic_table [id=89052c0e7bdc4b8e89824733ef19dd78]). Expected version 1 got 0
I20240501 20:09:56.592053 26954 tablet.cc:1722] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:56.596439 26957 tablet.cc:1722] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:56.597949 26957 tablet.cc:1620] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:56.599195 20826 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 7bbc34cad98b46468691ea4cdfc53e4f (table rebalancer_dynamic_table [id=89052c0e7bdc4b8e89824733ef19dd78]). Expected version 1 got 0
I20240501 20:09:56.600085 20818 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 7bbc34cad98b46468691ea4cdfc53e4f (table rebalancer_dynamic_table [id=89052c0e7bdc4b8e89824733ef19dd78]). Expected version 1 got 0
I20240501 20:09:56.605326 26954 tablet.cc:1620] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:56.622504 20826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=89052c0e7bdc4b8e89824733ef19dd78] alter complete (version 1)
I20240501 20:09:56.642024 26891 consensus_queue.cc:1035] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:56.661439 26888 consensus_queue.cc:1035] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:56.671175 26720 consensus_queue.cc:1035] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:56.679055 26891 consensus_queue.cc:1035] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:56.726009 26717 consensus_queue.cc:1035] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:56.733358 20819 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:56.743099 26953 tablet.cc:1722] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:56.743552 26954 tablet.cc:1722] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:56.743713 26953 tablet.cc:1620] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:56.744169 26954 tablet.cc:1620] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:09:56.749707 26957 tablet.cc:1722] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:56.750252 26957 tablet.cc:1620] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:56.763271 20819 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=89052c0e7bdc4b8e89824733ef19dd78] alter complete (version 2)
I20240501 20:09:56.789533 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:56.807171 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 8978a4c14f0a4446a979d3fc05248085 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:56.807493 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 8978a4c14f0a4446a979d3fc05248085 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:56.807801 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8978a4c14f0a4446a979d3fc05248085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.807960 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8978a4c14f0a4446a979d3fc05248085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.807991 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 25c9aac0679a41579f390c49f546f66b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:56.808394 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c9aac0679a41579f390c49f546f66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.811923 24090 tablet_service.cc:1467] Processing CreateTablet for tablet d75d2ef39a5347e2b6fbe3145c207280 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:56.812378 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75d2ef39a5347e2b6fbe3145c207280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.816083 26977 tablet_bootstrap.cc:492] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:56.816426 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 25c9aac0679a41579f390c49f546f66b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:56.816452 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 8978a4c14f0a4446a979d3fc05248085 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:56.816833 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c9aac0679a41579f390c49f546f66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.816900 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8978a4c14f0a4446a979d3fc05248085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.817955 26977 tablet_bootstrap.cc:654] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.823000 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 25c9aac0679a41579f390c49f546f66b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:56.823060 21076 tablet_service.cc:1467] Processing CreateTablet for tablet d75d2ef39a5347e2b6fbe3145c207280 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:56.823521 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75d2ef39a5347e2b6fbe3145c207280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.823668 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c9aac0679a41579f390c49f546f66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.826431 21207 tablet_service.cc:1467] Processing CreateTablet for tablet d75d2ef39a5347e2b6fbe3145c207280 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:56.826864 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75d2ef39a5347e2b6fbe3145c207280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:56.838348 26981 tablet_bootstrap.cc:492] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:56.842278 26981 tablet_bootstrap.cc:654] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.842780 26977 tablet_bootstrap.cc:492] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:56.843228 26977 ts_tablet_manager.cc:1367] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.013s
I20240501 20:09:56.848294 26977 raft_consensus.cc:348] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.848856 26977 raft_consensus.cc:374] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.849002 26977 raft_consensus.cc:729] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.849435 26977 consensus_queue.cc:260] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.852321 26981 tablet_bootstrap.cc:492] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:56.852484 26981 ts_tablet_manager.cc:1367] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240501 20:09:56.853487 26981 raft_consensus.cc:348] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.853698 26981 raft_consensus.cc:374] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.853776 26981 raft_consensus.cc:729] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.854065 26981 consensus_queue.cc:260] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.854683 26981 ts_tablet_manager.cc:1398] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:56.855084 26981 tablet_bootstrap.cc:492] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:56.857206 26981 tablet_bootstrap.cc:654] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.860108 26977 ts_tablet_manager.cc:1398] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 20:09:56.873884 26985 tablet_bootstrap.cc:492] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:56.876845 26980 tablet_bootstrap.cc:492] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:56.877660 26985 tablet_bootstrap.cc:654] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.881434 26978 tablet_bootstrap.cc:492] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:56.884315 26980 tablet_bootstrap.cc:654] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.884195 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:56.894033 26978 tablet_bootstrap.cc:654] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:56.904047 26954 tablet.cc:1722] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:56.910351 26957 tablet.cc:1722] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:56.935614 26985 tablet_bootstrap.cc:492] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:56.935972 26985 ts_tablet_manager.cc:1367] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.000s
I20240501 20:09:56.937361 26985 raft_consensus.cc:348] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:56.937783 26985 raft_consensus.cc:374] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.901867 26953 tablet.cc:1722] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:56.940793 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=89052c0e7bdc4b8e89824733ef19dd78] alter complete (version 3)
I20240501 20:09:56.949119 26985 raft_consensus.cc:729] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.950109 26985 consensus_queue.cc:260] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:56.952237 26985 ts_tablet_manager.cc:1398] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 20:09:56.957643 26981 tablet_bootstrap.cc:492] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:56.958030 26981 ts_tablet_manager.cc:1367] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.103s user 0.038s sys 0.034s
I20240501 20:09:56.959193 26981 raft_consensus.cc:348] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.960882 26981 raft_consensus.cc:374] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.961480 26980 tablet_bootstrap.cc:492] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:56.961555 26981 raft_consensus.cc:729] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.963155 26980 ts_tablet_manager.cc:1367] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.086s user 0.005s sys 0.000s
I20240501 20:09:56.964447 26980 raft_consensus.cc:348] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.962023 26981 consensus_queue.cc:260] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.972994 21208 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:56.985834 26980 raft_consensus.cc:374] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:56.986042 26980 raft_consensus.cc:729] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:56.986425 26980 consensus_queue.cc:260] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:56.988826 26981 ts_tablet_manager.cc:1398] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20240501 20:09:57.010766 26980 ts_tablet_manager.cc:1398] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20240501 20:09:57.011240 26980 tablet_bootstrap.cc:492] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:57.023556 26669 raft_consensus.cc:482] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:57.023818 26669 raft_consensus.cc:504] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.024766 26669 leader_election.cc:288] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:57.025831 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c9aac0679a41579f390c49f546f66b" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
W20240501 20:09:57.026669 21293 leader_election.cc:341] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:57.043380 20820 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:57.043609 20820 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:57.046873 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 7bbc34cad98b46468691ea4cdfc53e4f
I20240501 20:09:57.048219 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbc34cad98b46468691ea4cdfc53e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:57 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:57.048812 27000 tablet_replica.cc:330] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:57.048918 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbc34cad98b46468691ea4cdfc53e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:57 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:57.049124 27000 raft_consensus.cc:2229] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:57.049314 23689 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbc34cad98b46468691ea4cdfc53e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:57 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:09:57.049818 27000 raft_consensus.cc:2258] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:57.053483 27000 ts_tablet_manager.cc:1875] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:57.075392 23691 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:57.090749 27000 ts_tablet_manager.cc:1888] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:57.090929 27000 log.cc:1196] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/7bbc34cad98b46468691ea4cdfc53e4f
I20240501 20:09:57.090960 26980 tablet_bootstrap.cc:654] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:57.099059 27000 ts_tablet_manager.cc:1909] T 7bbc34cad98b46468691ea4cdfc53e4f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:09:57.102098 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 7bbc34cad98b46468691ea4cdfc53e4f (table rebalancer_dynamic_table_new_name [id=89052c0e7bdc4b8e89824733ef19dd78]) successfully deleted
I20240501 20:09:57.109802 27001 tablet_replica.cc:330] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:57.110065 27001 raft_consensus.cc:2229] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:57.110610 27001 raft_consensus.cc:2258] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:57.111230 26978 tablet_bootstrap.cc:492] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:57.111452 26978 ts_tablet_manager.cc:1367] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.230s user 0.009s sys 0.036s
I20240501 20:09:57.115005 26978 raft_consensus.cc:348] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.115347 26978 raft_consensus.cc:374] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:57.115468 26978 raft_consensus.cc:729] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:57.115839 26978 consensus_queue.cc:260] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.116468 26978 ts_tablet_manager.cc:1398] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:57.116839 26978 tablet_bootstrap.cc:492] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:57.118733 26978 tablet_bootstrap.cc:654] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:57.125742 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c9aac0679a41579f390c49f546f66b" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:57.126014 23977 raft_consensus.cc:2454] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:09:57.131731 27001 ts_tablet_manager.cc:1875] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:57.134202 21291 leader_election.cc:302] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 744c5a532e724391830c0785d32eb2fb; no voters: 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:57.137861 27002 tablet_replica.cc:330] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:09:57.138134 27002 raft_consensus.cc:2229] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:57.138228 24565 raft_consensus.cc:2781] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:57.138345 24565 raft_consensus.cc:482] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:57.138391 27002 raft_consensus.cc:2258] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:57.138456 24565 raft_consensus.cc:3037] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.141629 24565 raft_consensus.cc:504] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.142625 24565 leader_election.cc:288] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:57.143242 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c9aac0679a41579f390c49f546f66b" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
W20240501 20:09:57.146220 21293 leader_election.cc:341] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:57.149583 26745 raft_consensus.cc:482] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:57.149724 26745 raft_consensus.cc:504] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.150158 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:57.150451 26745 leader_election.cc:288] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:57.152309 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
W20240501 20:09:57.152988 21034 leader_election.cc:341] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:57.155143 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
W20240501 20:09:57.156005 21033 leader_election.cc:341] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:57.156167 21033 leader_election.cc:302] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feac73db0a3b4aafa374ca5dfd322b08; no voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:57.157292 26745 raft_consensus.cc:2726] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:57.159193 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c9aac0679a41579f390c49f546f66b" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:57.159449 23977 raft_consensus.cc:3037] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.163496 23977 raft_consensus.cc:2454] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:09:57.165763 21336 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:09:57.177727 20945 tablet_service.cc:1467] Processing CreateTablet for tablet d9ec76693c1041ac88a7f5962239b116 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f95a965b4364b0193ed00805f56d5f2]), partition=
I20240501 20:09:57.177989 24088 tablet_service.cc:1467] Processing CreateTablet for tablet d9ec76693c1041ac88a7f5962239b116 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f95a965b4364b0193ed00805f56d5f2]), partition=
I20240501 20:09:57.178537 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ec76693c1041ac88a7f5962239b116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:57.183734 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ec76693c1041ac88a7f5962239b116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:57.197312 21291 leader_election.cc:302] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 744c5a532e724391830c0785d32eb2fb; no voters: 8559229e044b4e1ea661b881b3f9af50
I20240501 20:09:57.198468 26669 raft_consensus.cc:2781] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:57.198697 26669 raft_consensus.cc:686] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:09:57.199179 26669 consensus_queue.cc:237] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.207052 26990 raft_consensus.cc:482] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:57.207356 26990 raft_consensus.cc:504] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.208586 26990 leader_election.cc:288] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:57.210573 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8978a4c14f0a4446a979d3fc05248085" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:57.210847 24108 raft_consensus.cc:2454] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:57.210987 26978 tablet_bootstrap.cc:492] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:57.211144 26978 ts_tablet_manager.cc:1367] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.094s user 0.008s sys 0.010s
I20240501 20:09:57.212373 26978 raft_consensus.cc:348] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.212787 23957 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:57.212782 21163 leader_election.cc:302] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:57.213347 26629 raft_consensus.cc:2781] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:57.213440 26629 raft_consensus.cc:482] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:57.213533 26629 raft_consensus.cc:3037] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.213631 26978 raft_consensus.cc:374] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:57.213825 26978 raft_consensus.cc:729] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:57.214298 26978 consensus_queue.cc:260] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.215205 26978 ts_tablet_manager.cc:1398] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:57.216069 26978 tablet_bootstrap.cc:492] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:57.217376 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8978a4c14f0a4446a979d3fc05248085" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:57.217617 23975 raft_consensus.cc:2454] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:57.218423 27001 ts_tablet_manager.cc:1888] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:57.218550 27001 log.cc:1196] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/7bbc34cad98b46468691ea4cdfc53e4f
I20240501 20:09:57.219497 26629 raft_consensus.cc:504] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.220496 26978 tablet_bootstrap.cc:654] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:57.220899 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8978a4c14f0a4446a979d3fc05248085" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:57.221107 23975 raft_consensus.cc:3037] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.220727 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8978a4c14f0a4446a979d3fc05248085" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:57.222107 24109 raft_consensus.cc:3037] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.224030 23975 raft_consensus.cc:2454] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:57.224823 21161 leader_election.cc:302] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:57.225047 24109 raft_consensus.cc:2454] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:57.225073 26322 raft_consensus.cc:2781] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:57.225173 26322 raft_consensus.cc:686] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:57.225417 26322 consensus_queue.cc:237] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.226641 26629 leader_election.cc:288] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:57.232717 26978 tablet_bootstrap.cc:492] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:57.233435 26978 ts_tablet_manager.cc:1367] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20240501 20:09:57.235661 26978 raft_consensus.cc:348] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.236431 26978 raft_consensus.cc:374] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:57.236663 26978 raft_consensus.cc:729] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:57.237185 26978 consensus_queue.cc:260] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.239279 26978 ts_tablet_manager.cc:1398] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:57.240432 26978 tablet_bootstrap.cc:492] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:57.240731 27001 ts_tablet_manager.cc:1909] T 7bbc34cad98b46468691ea4cdfc53e4f P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:57.246856 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 7bbc34cad98b46468691ea4cdfc53e4f (table rebalancer_dynamic_table_new_name [id=89052c0e7bdc4b8e89824733ef19dd78]) successfully deleted
I20240501 20:09:57.248345 26978 tablet_bootstrap.cc:654] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:57.277426 26980 tablet_bootstrap.cc:492] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:57.277599 26980 ts_tablet_manager.cc:1367] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.266s user 0.004s sys 0.000s
I20240501 20:09:57.278635 26980 raft_consensus.cc:348] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.278892 26980 raft_consensus.cc:374] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:57.278957 26980 raft_consensus.cc:729] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:57.279206 26980 consensus_queue.cc:260] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.285436 26980 ts_tablet_manager.cc:1398] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:09:57.289878 20947 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:57.169328 27002 ts_tablet_manager.cc:1875] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:57.340466 24088 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:57.344111 27019 tablet_bootstrap.cc:492] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
W20240501 20:09:57.367363 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 170075us
I20240501 20:09:57.370632 21336 tablet_service.cc:1467] Processing CreateTablet for tablet d9ec76693c1041ac88a7f5962239b116 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f95a965b4364b0193ed00805f56d5f2]), partition=
I20240501 20:09:57.371145 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ec76693c1041ac88a7f5962239b116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:57.374505 20820 catalog_manager.cc:5484] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:57.375385 27019 tablet_bootstrap.cc:654] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:57.391111 20820 catalog_manager.cc:5484] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:57.396917 23824 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:09:57.401335 21076 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:09:57.432638 27002 ts_tablet_manager.cc:1888] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:57.433743 27019 tablet_bootstrap.cc:492] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:57.433996 27019 ts_tablet_manager.cc:1367] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
I20240501 20:09:57.434756 27019 raft_consensus.cc:348] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:57.435102 27019 raft_consensus.cc:374] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:57.435197 27019 raft_consensus.cc:729] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:57.435472 27019 consensus_queue.cc:260] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:57.436193 27019 ts_tablet_manager.cc:1398] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:57.436399 27002 log.cc:1196] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/7bbc34cad98b46468691ea4cdfc53e4f
I20240501 20:09:57.437021 27002 ts_tablet_manager.cc:1909] T 7bbc34cad98b46468691ea4cdfc53e4f P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:09:57.442325 26978 tablet_bootstrap.cc:492] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:57.443578 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 7bbc34cad98b46468691ea4cdfc53e4f (table rebalancer_dynamic_table_new_name [id=89052c0e7bdc4b8e89824733ef19dd78]) successfully deleted
I20240501 20:09:57.444644 27023 tablet_bootstrap.cc:492] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:57.445457 26978 ts_tablet_manager.cc:1367] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.205s user 0.022s sys 0.065s
I20240501 20:09:57.447929 26978 raft_consensus.cc:348] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:57.450464 27023 tablet_bootstrap.cc:654] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:57.451668 26978 raft_consensus.cc:374] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:57.451769 26978 raft_consensus.cc:729] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:57.452013 26978 consensus_queue.cc:260] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:57.452553 26978 ts_tablet_manager.cc:1398] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 20:09:57.476820 27023 tablet_bootstrap.cc:492] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:57.477075 27023 ts_tablet_manager.cc:1367] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.004s
I20240501 20:09:57.477811 27023 raft_consensus.cc:348] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:57.478118 27023 raft_consensus.cc:374] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:57.478217 27023 raft_consensus.cc:729] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:57.478492 27023 consensus_queue.cc:260] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:57.479414 27023 ts_tablet_manager.cc:1398] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:57.516769 26474 raft_consensus.cc:482] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:57.516949 26474 raft_consensus.cc:504] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:57.517794 26474 leader_election.cc:288] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:57.518719 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c9aac0679a41579f390c49f546f66b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:57.524302 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c9aac0679a41579f390c49f546f66b" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:57.524626 23975 raft_consensus.cc:2379] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8559229e044b4e1ea661b881b3f9af50 in current term 1: Already voted for candidate 61380023001a4ee9ada9ec2c6c1b7159 in this term.
I20240501 20:09:57.525061 24043 leader_election.cc:302] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50; no voters: 61380023001a4ee9ada9ec2c6c1b7159, 744c5a532e724391830c0785d32eb2fb
I20240501 20:09:57.525682 26474 raft_consensus.cc:3037] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.530512 26474 raft_consensus.cc:2726] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:57.573796 26474 raft_consensus.cc:482] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:57.573989 26474 raft_consensus.cc:504] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.574750 26474 leader_election.cc:288] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:57.575605 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:57.575801 21095 raft_consensus.cc:2454] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:57.576822 26322 raft_consensus.cc:482] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:57.576963 26322 raft_consensus.cc:504] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.577701 26322 leader_election.cc:288] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:57.578210 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:57.578411 21095 raft_consensus.cc:2454] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:57.578781 21161 leader_election.cc:302] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:57.579988 26990 raft_consensus.cc:2781] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:57.580063 26990 raft_consensus.cc:482] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:57.580116 26990 raft_consensus.cc:3037] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.582897 26990 raft_consensus.cc:504] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.583925 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:57.584090 21095 raft_consensus.cc:3037] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.586869 21095 raft_consensus.cc:2454] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:57.587625 21161 leader_election.cc:302] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:57.588050 26629 raft_consensus.cc:2781] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:57.588145 26629 raft_consensus.cc:686] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:57.588393 26629 consensus_queue.cc:237] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.579530 24043 leader_election.cc:302] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:57.579571 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:57.596081 26224 raft_consensus.cc:2781] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:57.596266 26224 raft_consensus.cc:482] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:57.596429 26224 raft_consensus.cc:3037] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.598416 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:57.599195 26224 raft_consensus.cc:504] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:57.599689 24109 raft_consensus.cc:2379] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4af096f5a6a9440a94f6682b399efe20 in current term 1: Already voted for candidate 8559229e044b4e1ea661b881b3f9af50 in this term.
I20240501 20:09:57.600075 26224 leader_election.cc:288] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:57.602687 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:57.603125 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:57.603502 21098 raft_consensus.cc:2379] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8559229e044b4e1ea661b881b3f9af50 in current term 1: Already voted for candidate 4af096f5a6a9440a94f6682b399efe20 in this term.
I20240501 20:09:57.603794 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:57.604066 24109 raft_consensus.cc:2379] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4af096f5a6a9440a94f6682b399efe20 in current term 1: Already voted for candidate 8559229e044b4e1ea661b881b3f9af50 in this term.
I20240501 20:09:57.604441 26224 raft_consensus.cc:482] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:57.604629 26224 raft_consensus.cc:504] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:57.605804 26224 leader_election.cc:288] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:57.607198 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ec76693c1041ac88a7f5962239b116" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:57.607424 20965 raft_consensus.cc:2454] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:57.604759 24046 leader_election.cc:302] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:09:57.608734 20820 catalog_manager.cc:5484] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:57.609545 24043 leader_election.cc:302] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:57.609855 26474 raft_consensus.cc:2726] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:09:57.610050 26474 raft_consensus.cc:2781] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:57.610157 26474 raft_consensus.cc:482] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:57.610244 26474 raft_consensus.cc:3037] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.613180 26990 leader_election.cc:288] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:57.615593 26474 raft_consensus.cc:504] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:57.617064 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ec76693c1041ac88a7f5962239b116" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:57.617290 20965 raft_consensus.cc:3037] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.620189 27014 consensus_queue.cc:1035] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:57.620445 20965 raft_consensus.cc:2454] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:57.620893 24043 leader_election.cc:302] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:57.621397 26474 leader_election.cc:288] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:57.621964 26224 raft_consensus.cc:2781] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:57.644403 26224 raft_consensus.cc:686] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:57.646128 26224 consensus_queue.cc:237] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:57.651618 26990 consensus_queue.cc:1035] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:57.658367 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ec76693c1041ac88a7f5962239b116" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:57.658648 21357 raft_consensus.cc:2454] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:57.659061 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ec76693c1041ac88a7f5962239b116" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:57.659322 21358 raft_consensus.cc:3037] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:57.661300 27014 consensus_queue.cc:1035] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:57.662035 21358 raft_consensus.cc:2454] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:57.667249 20820 catalog_manager.cc:5484] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:57.677284 26629 consensus_queue.cc:1035] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:57.708782 20965 consensus_queue.cc:237] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:57.711519 23710 raft_consensus.cc:1261] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:57.712438 21358 raft_consensus.cc:1261] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:57.712392 27055 consensus_queue.cc:1035] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:57.712949 26375 consensus_queue.cc:1035] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:57.715313 27055 raft_consensus.cc:2932] T 7a2f74307a194dc49d184d4ea288b36d P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:57.716917 23710 raft_consensus.cc:2932] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:57.717234 21358 raft_consensus.cc:2932] T 7a2f74307a194dc49d184d4ea288b36d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:57.723832 20821 catalog_manager.cc:5484] T 7a2f74307a194dc49d184d4ea288b36d P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:57.729883 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2f74307a194dc49d184d4ea288b36d with delete_type TABLET_DATA_TOMBSTONED (TS 4af096f5a6a9440a94f6682b399efe20 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:57.730528 27060 tablet_replica.cc:330] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:57.730813 27060 raft_consensus.cc:2229] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:57.731070 27060 raft_consensus.cc:2258] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:57.731729 27060 ts_tablet_manager.cc:1875] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:57.737936 27060 ts_tablet_manager.cc:1888] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:09:57.738104 27060 log.cc:1196] T 7a2f74307a194dc49d184d4ea288b36d P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/7a2f74307a194dc49d184d4ea288b36d
I20240501 20:09:57.739056 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 7a2f74307a194dc49d184d4ea288b36d (table rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]) successfully deleted
I20240501 20:09:57.857743 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:57.872591 20965 raft_consensus.cc:1261] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:57.872663 21358 raft_consensus.cc:1261] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:57.873769 27028 consensus_queue.cc:1035] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:57.878144 26224 consensus_queue.cc:1035] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:57.909328 27102 tablet.cc:1722] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:57.910089 27102 tablet.cc:1620] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:57.919158 20823 catalog_manager.cc:5576] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399) does not have the latest schema for tablet d9ec76693c1041ac88a7f5962239b116 (table rebalancer_dynamic_table [id=7f95a965b4364b0193ed00805f56d5f2]). Expected version 1 got 0
I20240501 20:09:57.920720 20817 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet d9ec76693c1041ac88a7f5962239b116 (table rebalancer_dynamic_table [id=7f95a965b4364b0193ed00805f56d5f2]). Expected version 1 got 0
I20240501 20:09:57.921347 27104 tablet.cc:1722] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:57.926589 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=7f95a965b4364b0193ed00805f56d5f2] alter complete (version 1)
I20240501 20:09:57.932049 20823 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet d9ec76693c1041ac88a7f5962239b116 (table rebalancer_dynamic_table [id=7f95a965b4364b0193ed00805f56d5f2]). Expected version 1 got 0
I20240501 20:09:57.936262 27104 tablet.cc:1620] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:57.941766 20817 catalog_manager.cc:5576] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399) does not have the latest schema for tablet d9ec76693c1041ac88a7f5962239b116 (table rebalancer_dynamic_table [id=7f95a965b4364b0193ed00805f56d5f2]). Expected version 1 got 0
I20240501 20:09:57.941764 27103 tablet.cc:1722] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:57.942754 27103 tablet.cc:1620] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: MemRowSet was empty: no flush needed.
I20240501 20:09:57.974395 26990 consensus_queue.cc:1035] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:57.987733 26629 consensus_queue.cc:1035] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:58.042140 20819 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:58.059384 27102 tablet.cc:1722] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:58.059901 27102 tablet.cc:1620] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:58.060336 27104 tablet.cc:1722] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:58.060747 27104 tablet.cc:1620] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:58.061602 27103 tablet.cc:1722] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:58.062069 27103 tablet.cc:1620] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: MemRowSet was empty: no flush needed.
I20240501 20:09:58.075724 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=7f95a965b4364b0193ed00805f56d5f2] alter complete (version 2)
I20240501 20:09:58.133457 21208 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:09:58.176654 23911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 120673us
I20240501 20:09:58.183452 20818 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:58.197989 27102 tablet.cc:1722] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:58.200733 21336 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 20:09:58.206012 27104 tablet.cc:1722] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:58.222435 23689 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:58.233630 20818 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=7f95a965b4364b0193ed00805f56d5f2] alter complete (version 3)
I20240501 20:09:58.244762 27103 tablet.cc:1722] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:58.249505 23957 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:58.285830 24088 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:09:58.292207 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:58.303205 21336 tablet_service.cc:1467] Processing CreateTablet for tablet c018d1b99ad749e59a2cf338001d38e4 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:58.303709 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c018d1b99ad749e59a2cf338001d38e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.303784 21076 tablet_service.cc:1467] Processing CreateTablet for tablet c018d1b99ad749e59a2cf338001d38e4 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:58.304229 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c018d1b99ad749e59a2cf338001d38e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.304334 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 719e9466ddd347e8810c3d631a770740 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:58.304730 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719e9466ddd347e8810c3d631a770740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.308338 20945 tablet_service.cc:1467] Processing CreateTablet for tablet c018d1b99ad749e59a2cf338001d38e4 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:58.308737 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c018d1b99ad749e59a2cf338001d38e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.317076 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 719e9466ddd347e8810c3d631a770740 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:58.317268 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 719e9466ddd347e8810c3d631a770740 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:58.317517 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719e9466ddd347e8810c3d631a770740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.317665 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719e9466ddd347e8810c3d631a770740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.321429 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca79fa39dae493789d9ddf79a7d54f0 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:58.321846 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca79fa39dae493789d9ddf79a7d54f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.322007 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca79fa39dae493789d9ddf79a7d54f0 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:58.322451 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca79fa39dae493789d9ddf79a7d54f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.330235 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca79fa39dae493789d9ddf79a7d54f0 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:58.330643 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca79fa39dae493789d9ddf79a7d54f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.336997 20823 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:58.337267 20823 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:58.337438 23826 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:09:58.344441 20823 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet d9ec76693c1041ac88a7f5962239b116
I20240501 20:09:58.345374 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ec76693c1041ac88a7f5962239b116 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:58 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:58.345783 21337 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ec76693c1041ac88a7f5962239b116 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:58 UTC) from {username='slave'} at 127.0.0.1:42940
I20240501 20:09:58.373828 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ec76693c1041ac88a7f5962239b116 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:58 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:09:58.386140 27136 tablet_bootstrap.cc:492] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:58.386788 27140 tablet_replica.cc:330] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:58.387167 27141 tablet_replica.cc:330] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:09:58.410207 27139 tablet_bootstrap.cc:492] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:58.411028 27134 tablet_bootstrap.cc:492] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:58.414045 27141 raft_consensus.cc:2229] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:58.414515 27140 raft_consensus.cc:2229] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:58.417090 27141 raft_consensus.cc:2258] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:58.420857 27138 tablet_bootstrap.cc:492] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:58.421149 27140 raft_consensus.cc:2258] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 20:09:58.422158 23647 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 123413us
I20240501 20:09:58.423815 27141 ts_tablet_manager.cc:1875] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:58.420361 27137 tablet_bootstrap.cc:492] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:58.419602 20947 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:58.425426 27140 ts_tablet_manager.cc:1875] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:58.427021 27134 tablet_bootstrap.cc:654] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.443516 27145 tablet_replica.cc:330] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:09:58.443950 27145 raft_consensus.cc:2229] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:58.444298 27145 raft_consensus.cc:2258] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:58.444980 27139 tablet_bootstrap.cc:654] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.445674 27145 ts_tablet_manager.cc:1875] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:58.446900 20823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:58.456573 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2790c81041854194922e7c1abd663e9a]), partition=
I20240501 20:09:58.457033 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.458283 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2790c81041854194922e7c1abd663e9a]), partition=
I20240501 20:09:58.459897 27133 tablet_bootstrap.cc:492] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:58.461370 27133 tablet_bootstrap.cc:654] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.462534 27136 tablet_bootstrap.cc:654] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.464043 27138 tablet_bootstrap.cc:654] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.465304 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.467154 27137 tablet_bootstrap.cc:654] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.468389 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2790c81041854194922e7c1abd663e9a]), partition=
I20240501 20:09:58.468677 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.486830 27133 tablet_bootstrap.cc:492] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:58.492071 27139 tablet_bootstrap.cc:492] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:58.492131 21076 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:09:58.492892 27133 ts_tablet_manager.cc:1367] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20240501 20:09:58.494050 27133 raft_consensus.cc:348] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:58.492261 27139 ts_tablet_manager.cc:1367] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.003s
I20240501 20:09:58.497439 27139 raft_consensus.cc:348] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:58.497743 27139 raft_consensus.cc:374] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.497876 27139 raft_consensus.cc:729] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.498117 27139 consensus_queue.cc:260] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:58.498719 27139 ts_tablet_manager.cc:1398] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:58.499006 27139 tablet_bootstrap.cc:492] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:58.500754 27139 tablet_bootstrap.cc:654] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.503326 27133 raft_consensus.cc:374] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.503463 27133 raft_consensus.cc:729] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.503847 27133 consensus_queue.cc:260] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:58.504909 27133 ts_tablet_manager.cc:1398] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 20:09:58.511329 27140 ts_tablet_manager.cc:1888] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:58.511447 27140 log.cc:1196] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/d9ec76693c1041ac88a7f5962239b116
I20240501 20:09:58.511919 27140 ts_tablet_manager.cc:1909] T d9ec76693c1041ac88a7f5962239b116 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:58.512907 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet d9ec76693c1041ac88a7f5962239b116 (table rebalancer_dynamic_table_new_name [id=7f95a965b4364b0193ed00805f56d5f2]) successfully deleted
I20240501 20:09:58.496294 27141 ts_tablet_manager.cc:1888] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:58.517988 27141 log.cc:1196] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/d9ec76693c1041ac88a7f5962239b116
I20240501 20:09:58.518402 27141 ts_tablet_manager.cc:1909] T d9ec76693c1041ac88a7f5962239b116 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting consensus metadata
I20240501 20:09:58.520303 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet d9ec76693c1041ac88a7f5962239b116 (table rebalancer_dynamic_table_new_name [id=7f95a965b4364b0193ed00805f56d5f2]) successfully deleted
I20240501 20:09:58.524086 27136 tablet_bootstrap.cc:492] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:58.524303 27136 ts_tablet_manager.cc:1367] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.138s user 0.007s sys 0.000s
I20240501 20:09:58.525095 27136 raft_consensus.cc:348] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.525365 27136 raft_consensus.cc:374] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.525491 27136 raft_consensus.cc:729] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.525753 27136 consensus_queue.cc:260] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.526327 27136 ts_tablet_manager.cc:1398] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:58.526602 27136 tablet_bootstrap.cc:492] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:09:58.528213 27136 tablet_bootstrap.cc:654] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.530443 27137 tablet_bootstrap.cc:492] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:58.530622 27137 ts_tablet_manager.cc:1367] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.000s
I20240501 20:09:58.533850 27137 raft_consensus.cc:348] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:58.535286 27137 raft_consensus.cc:374] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.535492 27137 raft_consensus.cc:729] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.535952 27137 consensus_queue.cc:260] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:58.541409 27137 ts_tablet_manager.cc:1398] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 20:09:58.541880 27150 tablet_bootstrap.cc:492] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:58.543561 27150 tablet_bootstrap.cc:654] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.544731 27145 ts_tablet_manager.cc:1888] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:58.548048 27145 log.cc:1196] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/d9ec76693c1041ac88a7f5962239b116
I20240501 20:09:58.548704 27145 ts_tablet_manager.cc:1909] T d9ec76693c1041ac88a7f5962239b116 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:09:58.550338 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet d9ec76693c1041ac88a7f5962239b116 (table rebalancer_dynamic_table_new_name [id=7f95a965b4364b0193ed00805f56d5f2]) successfully deleted
I20240501 20:09:58.551268 27137 tablet_bootstrap.cc:492] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:58.553428 27137 tablet_bootstrap.cc:654] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.556454 27138 tablet_bootstrap.cc:492] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:58.556687 27138 ts_tablet_manager.cc:1367] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.136s user 0.005s sys 0.005s
I20240501 20:09:58.557598 27138 raft_consensus.cc:348] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:58.557859 27138 raft_consensus.cc:374] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.557973 27138 raft_consensus.cc:729] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.558262 27138 consensus_queue.cc:260] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:58.558843 27138 ts_tablet_manager.cc:1398] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:58.563073 26387 raft_consensus.cc:482] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:58.563254 26387 raft_consensus.cc:504] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.564180 26387 leader_election.cc:288] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:09:58.565176 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca79fa39dae493789d9ddf79a7d54f0" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:09:58.565763 27134 tablet_bootstrap.cc:492] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:58.565932 27134 ts_tablet_manager.cc:1367] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.155s user 0.010s sys 0.000s
I20240501 20:09:58.566078 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca79fa39dae493789d9ddf79a7d54f0" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:09:58.566313 23911 leader_election.cc:341] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:58.566725 23912 leader_election.cc:341] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:58.566823 23912 leader_election.cc:302] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:09:58.566802 27134 raft_consensus.cc:348] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:58.567008 27134 raft_consensus.cc:374] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.567080 27134 raft_consensus.cc:729] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.567318 27134 consensus_queue.cc:260] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:58.567507 26387 raft_consensus.cc:2726] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:58.568045 27134 ts_tablet_manager.cc:1398] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:58.568400 27134 tablet_bootstrap.cc:492] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:09:58.570272 27134 tablet_bootstrap.cc:654] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.572288 27139 tablet_bootstrap.cc:492] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:58.572476 27138 tablet_bootstrap.cc:492] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:58.573933 27138 tablet_bootstrap.cc:654] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:58.574167 27139 ts_tablet_manager.cc:1367] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.000s
I20240501 20:09:58.574980 27139 raft_consensus.cc:348] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.575387 27139 raft_consensus.cc:374] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.575589 27139 raft_consensus.cc:729] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.576169 27137 tablet_bootstrap.cc:492] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:58.577080 27137 ts_tablet_manager.cc:1367] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.005s
I20240501 20:09:58.577804 27137 raft_consensus.cc:348] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:58.578069 27136 tablet_bootstrap.cc:492] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:09:58.578305 27136 ts_tablet_manager.cc:1367] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20240501 20:09:58.579247 27136 raft_consensus.cc:348] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:58.579578 27136 raft_consensus.cc:374] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.579715 27136 raft_consensus.cc:729] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.580042 27136 consensus_queue.cc:260] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:58.581815 27136 ts_tablet_manager.cc:1398] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:58.582235 27139 consensus_queue.cc:260] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.582811 27137 raft_consensus.cc:374] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.582828 27139 ts_tablet_manager.cc:1398] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:09:58.582943 27137 raft_consensus.cc:729] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.583249 27137 consensus_queue.cc:260] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:58.584715 27150 tablet_bootstrap.cc:492] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:58.584901 27150 ts_tablet_manager.cc:1367] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.003s
I20240501 20:09:58.585553 27150 raft_consensus.cc:348] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:58.585803 27150 raft_consensus.cc:374] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.585889 27150 raft_consensus.cc:729] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.586134 27150 consensus_queue.cc:260] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:58.586354 27138 tablet_bootstrap.cc:492] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:58.586540 27138 ts_tablet_manager.cc:1367] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240501 20:09:58.587568 27138 raft_consensus.cc:348] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:58.587744 27150 ts_tablet_manager.cc:1398] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:58.587812 27138 raft_consensus.cc:374] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.587945 27138 raft_consensus.cc:729] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.588109 27137 ts_tablet_manager.cc:1398] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:09:58.588215 27138 consensus_queue.cc:260] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:58.591820 27138 ts_tablet_manager.cc:1398] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 20:09:58.598251 27134 tablet_bootstrap.cc:492] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:09:58.598521 27134 ts_tablet_manager.cc:1367] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240501 20:09:58.599442 27134 raft_consensus.cc:348] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.599774 27134 raft_consensus.cc:374] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:58.599890 27134 raft_consensus.cc:729] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:09:58.600191 27134 consensus_queue.cc:260] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.600796 27134 ts_tablet_manager.cc:1398] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:58.605633 27135 raft_consensus.cc:482] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:58.605775 27135 raft_consensus.cc:504] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:58.606457 27135 leader_election.cc:288] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:58.607023 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719e9466ddd347e8810c3d631a770740" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:58.607256 23846 raft_consensus.cc:2454] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:58.607997 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719e9466ddd347e8810c3d631a770740" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:58.608270 21095 raft_consensus.cc:2454] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:58.609380 23647 leader_election.cc:302] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:58.610103 27135 raft_consensus.cc:2781] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:58.610215 27135 raft_consensus.cc:482] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:58.610293 27135 raft_consensus.cc:3037] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.617921 27135 raft_consensus.cc:504] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:58.619668 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719e9466ddd347e8810c3d631a770740" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:58.619899 23846 raft_consensus.cc:3037] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.622879 27055 raft_consensus.cc:482] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:58.623028 27055 raft_consensus.cc:504] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:58.623167 23846 raft_consensus.cc:2454] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:58.623713 23647 leader_election.cc:302] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:58.623930 27055 leader_election.cc:288] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:58.624051 27021 raft_consensus.cc:2781] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:58.624182 27021 raft_consensus.cc:686] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:58.624496 27021 consensus_queue.cc:237] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:09:58.624584 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c018d1b99ad749e59a2cf338001d38e4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:58.624780 21358 raft_consensus.cc:2454] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:58.624940 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c018d1b99ad749e59a2cf338001d38e4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:58.625144 21095 raft_consensus.cc:2454] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:58.625164 20901 leader_election.cc:302] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:58.625802 27055 raft_consensus.cc:2781] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:58.625890 27055 raft_consensus.cc:482] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:58.625954 27055 raft_consensus.cc:3037] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.628233 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719e9466ddd347e8810c3d631a770740" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:58.628409 21095 raft_consensus.cc:3037] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.628842 27055 raft_consensus.cc:504] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:58.629427 27055 leader_election.cc:288] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:58.630640 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c018d1b99ad749e59a2cf338001d38e4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:58.630802 21097 raft_consensus.cc:3037] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.631875 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c018d1b99ad749e59a2cf338001d38e4" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:58.632056 21358 raft_consensus.cc:3037] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.632055 21095 raft_consensus.cc:2454] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:58.633065 27135 leader_election.cc:288] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:09:58.634618 21358 raft_consensus.cc:2454] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:58.637822 20821 catalog_manager.cc:5484] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:09:58.640259 20901 leader_election.cc:302] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:58.640729 21097 raft_consensus.cc:2454] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:58.641651 27055 raft_consensus.cc:2781] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:58.642192 27055 raft_consensus.cc:686] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:58.642673 27055 consensus_queue.cc:237] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:09:58.649971 26990 raft_consensus.cc:482] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:58.650118 26990 raft_consensus.cc:504] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:58.650830 26990 leader_election.cc:288] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:58.651924 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b5eac6c24f48e0bfc1b14ec88e66a5" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:09:58.652119 20965 raft_consensus.cc:2454] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:58.652827 21161 leader_election.cc:302] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:58.653834 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b5eac6c24f48e0bfc1b14ec88e66a5" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:58.654083 23975 raft_consensus.cc:2454] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:58.655941 26990 raft_consensus.cc:2781] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:58.656026 26990 raft_consensus.cc:482] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:58.656091 26990 raft_consensus.cc:3037] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.665784 26990 raft_consensus.cc:504] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:58.666342 26990 leader_election.cc:288] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:09:58.667202 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b5eac6c24f48e0bfc1b14ec88e66a5" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:09:58.667440 20965 raft_consensus.cc:3037] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.667879 20821 catalog_manager.cc:5484] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:09:58.670779 20965 raft_consensus.cc:2454] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:58.673970 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b5eac6c24f48e0bfc1b14ec88e66a5" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:58.674274 23975 raft_consensus.cc:3037] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.679335 23975 raft_consensus.cc:2454] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:58.680718 21161 leader_election.cc:302] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:58.681205 26990 raft_consensus.cc:2781] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:58.681380 26990 raft_consensus.cc:686] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:58.681694 26990 consensus_queue.cc:237] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:09:58.688031 20821 catalog_manager.cc:5484] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:09:58.712118 23846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed53558d93214d34aedca8505870ec61"
dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38572
I20240501 20:09:58.712424 23846 raft_consensus.cc:595] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:58.756870 26720 raft_consensus.cc:979] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: : Instructing follower 4af096f5a6a9440a94f6682b399efe20 to start an election
I20240501 20:09:58.757073 26720 raft_consensus.cc:1067] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Signalling peer 4af096f5a6a9440a94f6682b399efe20 to start an election
I20240501 20:09:58.757898 21228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed53558d93214d34aedca8505870ec61"
dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
from {username='slave'} at 127.13.251.134:60949
I20240501 20:09:58.758102 21228 raft_consensus.cc:482] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:58.758205 21228 raft_consensus.cc:3037] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:58.760313 21228 raft_consensus.cc:504] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:58.761520 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:58.761516 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:58.761540 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:58.761713 21358 raft_consensus.cc:3037] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:58.761732 23846 raft_consensus.cc:3032] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:58.761739 24109 raft_consensus.cc:3037] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:58.761847 23846 raft_consensus.cc:729] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:09:58.762138 23846 consensus_queue.cc:260] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:58.762670 23846 raft_consensus.cc:3037] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:58.764323 24109 raft_consensus.cc:2454] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 2.
I20240501 20:09:58.764899 21358 raft_consensus.cc:2454] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 2.
I20240501 20:09:58.765369 21162 leader_election.cc:302] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:58.765743 26629 raft_consensus.cc:2781] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:58.765874 26629 raft_consensus.cc:686] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 2 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:58.765901 21228 leader_election.cc:288] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 2 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:09:58.766104 26629 consensus_queue.cc:237] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:58.768496 23846 raft_consensus.cc:2454] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 2.
I20240501 20:09:58.770728 20821 catalog_manager.cc:5484] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 1 to 2, leader changed from c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 2 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:58.837854 20823 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:58.851121 27021 raft_consensus.cc:482] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:58.851339 27021 raft_consensus.cc:504] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.852167 27021 leader_election.cc:288] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:58.853073 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca79fa39dae493789d9ddf79a7d54f0" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:09:58.853327 23975 raft_consensus.cc:2454] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:58.854561 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca79fa39dae493789d9ddf79a7d54f0" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:58.854772 23644 leader_election.cc:302] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:09:58.854787 23846 raft_consensus.cc:2454] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:09:58.856137 27021 raft_consensus.cc:2781] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:58.856262 27021 raft_consensus.cc:482] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:58.856393 27021 raft_consensus.cc:3037] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.860560 27021 raft_consensus.cc:504] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.861311 27021 leader_election.cc:288] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:58.861866 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca79fa39dae493789d9ddf79a7d54f0" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:09:58.862075 23975 raft_consensus.cc:3037] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.862571 23844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca79fa39dae493789d9ddf79a7d54f0" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:58.862809 23844 raft_consensus.cc:3037] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:58.865768 23844 raft_consensus.cc:2454] T 5ca79fa39dae493789d9ddf79a7d54f0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:58.869382 23647 leader_election.cc:302] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:09:58.869801 27131 raft_consensus.cc:2781] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:58.869971 27131 raft_consensus.cc:686] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:09:58.870286 27131 consensus_queue.cc:237] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:09:58.873255 23975 raft_consensus.cc:2454] T 5ca79fa39dae493789d9ddf79a7d54f0 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:09:58.905467 20821 catalog_manager.cc:5484] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:09:58.909204 20965 raft_consensus.cc:1261] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:58.910153 23975 raft_consensus.cc:1261] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:58.911068 26990 consensus_queue.cc:1035] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:58.916769 26629 consensus_queue.cc:1035] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:58.932250 20819 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (table rebalancer_dynamic_table [id=2790c81041854194922e7c1abd663e9a]). Expected version 1 got 0
I20240501 20:09:58.945748 20821 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (table rebalancer_dynamic_table [id=2790c81041854194922e7c1abd663e9a]). Expected version 1 got 0
I20240501 20:09:58.969275 27218 tablet.cc:1722] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:58.969944 27218 tablet.cc:1620] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:58.972684 27216 tablet.cc:1722] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:58.973338 27216 tablet.cc:1620] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:58.974159 27217 tablet.cc:1722] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:58.974840 27217 tablet.cc:1620] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:58.979744 20819 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (table rebalancer_dynamic_table [id=2790c81041854194922e7c1abd663e9a]). Expected version 1 got 0
I20240501 20:09:58.989863 21208 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:09:58.989842 20819 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=2790c81041854194922e7c1abd663e9a] alter complete (version 1)
I20240501 20:09:59.004902 20819 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (table rebalancer_dynamic_table [id=2790c81041854194922e7c1abd663e9a]). Expected version 1 got 0
I20240501 20:09:59.027033 23689 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:59.073019 27156 consensus_queue.cc:1035] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:59.075474 27055 consensus_queue.cc:1035] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:59.109689 27131 consensus_queue.cc:1035] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:59.113685 20819 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:59.128633 27216 tablet.cc:1722] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:59.129122 27216 tablet.cc:1620] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:59.134753 27218 tablet.cc:1722] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:59.135244 27218 tablet.cc:1620] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:09:59.139791 23957 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:59.141023 27217 tablet.cc:1722] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:59.141566 27217 tablet.cc:1620] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:09:59.147136 27131 consensus_queue.cc:1035] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:59.152832 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=2790c81041854194922e7c1abd663e9a] alter complete (version 2)
I20240501 20:09:59.169027 21357 raft_consensus.cc:1261] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:09:59.169677 26629 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:59.194339 20947 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:09:59.205096 24109 raft_consensus.cc:1261] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:09:59.205624 26629 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:59.251629 23824 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:59.253787 23844 raft_consensus.cc:1261] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:09:59.254447 21336 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 20:09:59.255435 26990 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:59.263103 20821 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:59.277244 27131 consensus_queue.cc:1035] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:59.283679 27216 tablet.cc:1722] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240501 20:09:59.286691 21033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 145741us
I20240501 20:09:59.289579 27218 tablet.cc:1722] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:59.313069 24088 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:59.318284 27217 tablet.cc:1722] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:59.336195 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=2790c81041854194922e7c1abd663e9a] alter complete (version 3)
I20240501 20:09:59.358744 27135 consensus_queue.cc:1035] T 5ca79fa39dae493789d9ddf79a7d54f0 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:59.371448 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 75017.9us
I20240501 20:09:59.377300 21076 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:09:59.428776 20826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:59.441256 20819 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:59.441507 20819 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:59.443850 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 170ccc5b4ec949f1aa0652e43a3b9e5c (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.443871 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 170ccc5b4ec949f1aa0652e43a3b9e5c (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.444263 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170ccc5b4ec949f1aa0652e43a3b9e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.444311 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170ccc5b4ec949f1aa0652e43a3b9e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.452698 20819 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 35b5eac6c24f48e0bfc1b14ec88e66a5
I20240501 20:09:59.453784 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:09:59.454701 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 170ccc5b4ec949f1aa0652e43a3b9e5c (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.455332 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170ccc5b4ec949f1aa0652e43a3b9e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.455917 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:09:59.455925 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 73fcc9ac412643088c996b76fbb1e680 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.456375 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73fcc9ac412643088c996b76fbb1e680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.460027 21206 tablet_service.cc:1467] Processing CreateTablet for tablet 73fcc9ac412643088c996b76fbb1e680 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.460575 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73fcc9ac412643088c996b76fbb1e680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.460696 21336 tablet_service.cc:1467] Processing CreateTablet for tablet b7eb4c409a2945f0b39bebd37fec4ea7 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.461099 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7eb4c409a2945f0b39bebd37fec4ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.464659 21205 tablet_service.cc:1467] Processing CreateTablet for tablet b7eb4c409a2945f0b39bebd37fec4ea7 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.465057 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7eb4c409a2945f0b39bebd37fec4ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.466120 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:09:59.466648 20946 tablet_service.cc:1467] Processing CreateTablet for tablet b7eb4c409a2945f0b39bebd37fec4ea7 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.466698 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 73fcc9ac412643088c996b76fbb1e680 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.467015 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7eb4c409a2945f0b39bebd37fec4ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.467260 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73fcc9ac412643088c996b76fbb1e680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.496439 27261 tablet_bootstrap.cc:492] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:59.496953 27260 tablet_replica.cc:330] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:09:59.498255 27262 tablet_bootstrap.cc:492] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:09:59.503834 27260 raft_consensus.cc:2229] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:59.504112 27258 tablet_replica.cc:330] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:09:59.505296 27258 raft_consensus.cc:2229] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:59.507035 27260 raft_consensus.cc:2258] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.507566 27258 raft_consensus.cc:2258] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.508531 27258 ts_tablet_manager.cc:1875] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.512238 27259 tablet_bootstrap.cc:492] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:59.515313 27260 ts_tablet_manager.cc:1875] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.516068 27257 tablet_replica.cc:330] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:09:59.520090 27262 tablet_bootstrap.cc:654] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.521577 27257 raft_consensus.cc:2229] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:59.521683 27261 tablet_bootstrap.cc:654] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.522295 27257 raft_consensus.cc:2258] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.540084 27256 tablet_bootstrap.cc:492] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:09:59.542271 27261 tablet_bootstrap.cc:492] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:59.542515 27261 ts_tablet_manager.cc:1367] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.007s
I20240501 20:09:59.543651 27261 raft_consensus.cc:348] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.544011 27261 raft_consensus.cc:374] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.544011 27258 ts_tablet_manager.cc:1888] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:59.544137 27258 log.cc:1196] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/35b5eac6c24f48e0bfc1b14ec88e66a5
I20240501 20:09:59.544245 27261 raft_consensus.cc:729] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.544569 27260 ts_tablet_manager.cc:1888] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:59.544582 27261 consensus_queue.cc:260] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.544867 27260 log.cc:1196] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/35b5eac6c24f48e0bfc1b14ec88e66a5
I20240501 20:09:59.545640 27256 tablet_bootstrap.cc:654] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.546154 27260 ts_tablet_manager.cc:1909] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:09:59.546558 27261 ts_tablet_manager.cc:1398] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:59.548442 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (table rebalancer_dynamic_table_new_name [id=2790c81041854194922e7c1abd663e9a]) successfully deleted
I20240501 20:09:59.549702 27261 tablet_bootstrap.cc:492] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:09:59.553218 27256 tablet_bootstrap.cc:492] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:09:59.554322 27256 ts_tablet_manager.cc:1367] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240501 20:09:59.555629 27262 tablet_bootstrap.cc:492] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:09:59.555859 27262 ts_tablet_manager.cc:1367] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.009s
I20240501 20:09:59.556571 27261 tablet_bootstrap.cc:654] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.556972 27262 raft_consensus.cc:348] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.559217 27262 raft_consensus.cc:374] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.559350 27262 raft_consensus.cc:729] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.559724 27262 consensus_queue.cc:260] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.561133 27259 tablet_bootstrap.cc:654] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.562845 27258 ts_tablet_manager.cc:1909] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:09:59.564016 27257 ts_tablet_manager.cc:1875] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.566452 27270 tablet_bootstrap.cc:492] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:59.566643 27262 ts_tablet_manager.cc:1398] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:09:59.567260 27256 raft_consensus.cc:348] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.567503 27256 raft_consensus.cc:374] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.569546 27256 raft_consensus.cc:729] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.569978 27256 consensus_queue.cc:260] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.570173 27270 tablet_bootstrap.cc:654] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.572904 27261 tablet_bootstrap.cc:492] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:09:59.573230 27261 ts_tablet_manager.cc:1367] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240501 20:09:59.574015 27259 tablet_bootstrap.cc:492] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:59.575073 27256 ts_tablet_manager.cc:1398] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 20:09:59.575922 27261 raft_consensus.cc:348] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.576195 27261 raft_consensus.cc:374] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.576260 27261 raft_consensus.cc:729] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.576470 27261 consensus_queue.cc:260] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.576961 27261 ts_tablet_manager.cc:1398] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:09:59.577311 20826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:59.578390 27270 tablet_bootstrap.cc:492] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:59.578550 27270 ts_tablet_manager.cc:1367] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 20:09:59.578810 27259 ts_tablet_manager.cc:1367] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20240501 20:09:59.579308 27270 raft_consensus.cc:348] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.579512 27270 raft_consensus.cc:374] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.579669 27270 raft_consensus.cc:729] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.579727 27259 raft_consensus.cc:348] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.579959 27259 raft_consensus.cc:374] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.580025 27259 raft_consensus.cc:729] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.580332 27259 consensus_queue.cc:260] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.583295 27270 consensus_queue.cc:260] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.584424 27270 ts_tablet_manager.cc:1398] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:59.584703 27270 tablet_bootstrap.cc:492] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:59.587414 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (table rebalancer_dynamic_table_new_name [id=2790c81041854194922e7c1abd663e9a]) successfully deleted
I20240501 20:09:59.592455 23824 tablet_service.cc:1467] Processing CreateTablet for tablet eb552d92b68a40b4b35c995c98291c91 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=200a5ec20e86403bab80c66e8dd8da14]), partition=
I20240501 20:09:59.592893 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb552d92b68a40b4b35c995c98291c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.595933 24089 tablet_service.cc:1467] Processing CreateTablet for tablet eb552d92b68a40b4b35c995c98291c91 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=200a5ec20e86403bab80c66e8dd8da14]), partition=
I20240501 20:09:59.596228 27270 tablet_bootstrap.cc:654] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.596329 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb552d92b68a40b4b35c995c98291c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.596626 27156 raft_consensus.cc:482] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:59.596760 27156 raft_consensus.cc:504] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.597620 27156 leader_election.cc:288] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:59.598522 27259 ts_tablet_manager.cc:1398] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 20:09:59.600168 21206 tablet_service.cc:1467] Processing CreateTablet for tablet eb552d92b68a40b4b35c995c98291c91 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=200a5ec20e86403bab80c66e8dd8da14]), partition=
I20240501 20:09:59.601271 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73fcc9ac412643088c996b76fbb1e680" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:59.601490 24109 raft_consensus.cc:2454] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:59.602452 20902 leader_election.cc:302] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:59.602700 27156 raft_consensus.cc:2781] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:59.602773 27156 raft_consensus.cc:482] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:59.602833 27156 raft_consensus.cc:3037] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.605517 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73fcc9ac412643088c996b76fbb1e680" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:09:59.605928 27156 raft_consensus.cc:504] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.606171 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb552d92b68a40b4b35c995c98291c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.606571 27156 leader_election.cc:288] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
W20240501 20:09:59.607004 20903 leader_election.cc:341] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:59.607177 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73fcc9ac412643088c996b76fbb1e680" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:59.607440 24109 raft_consensus.cc:3037] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.609395 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73fcc9ac412643088c996b76fbb1e680" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
W20240501 20:09:59.612679 20903 leader_election.cc:341] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:59.613430 24109 raft_consensus.cc:2454] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:59.613999 20902 leader_election.cc:302] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:09:59.614463 27156 raft_consensus.cc:2781] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:59.614595 27156 raft_consensus.cc:686] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:59.614820 27156 consensus_queue.cc:237] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.622045 27259 tablet_bootstrap.cc:492] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:59.626045 27259 tablet_bootstrap.cc:654] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.628245 27278 tablet_bootstrap.cc:492] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:09:59.632964 27257 ts_tablet_manager.cc:1888] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:09:59.633154 27257 log.cc:1196] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/35b5eac6c24f48e0bfc1b14ec88e66a5
I20240501 20:09:59.633745 27257 ts_tablet_manager.cc:1909] T 35b5eac6c24f48e0bfc1b14ec88e66a5 P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:09:59.634857 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 35b5eac6c24f48e0bfc1b14ec88e66a5 (table rebalancer_dynamic_table_new_name [id=2790c81041854194922e7c1abd663e9a]) successfully deleted
I20240501 20:09:59.635864 27278 tablet_bootstrap.cc:654] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.635660 20826 catalog_manager.cc:5484] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:59.637957 27270 tablet_bootstrap.cc:492] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:59.638211 27270 ts_tablet_manager.cc:1367] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.000s
I20240501 20:09:59.638927 27270 raft_consensus.cc:348] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.639176 27270 raft_consensus.cc:374] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.639273 27270 raft_consensus.cc:729] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.639520 27270 consensus_queue.cc:260] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.640317 27270 ts_tablet_manager.cc:1398] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:59.640578 27278 tablet_bootstrap.cc:492] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:09:59.640736 27270 tablet_bootstrap.cc:492] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:09:59.640730 27278 ts_tablet_manager.cc:1367] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:09:59.641958 27278 raft_consensus.cc:348] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.642171 27278 raft_consensus.cc:374] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.642390 27278 raft_consensus.cc:729] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.642575 27270 tablet_bootstrap.cc:654] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.642629 27278 consensus_queue.cc:260] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.643195 27278 ts_tablet_manager.cc:1398] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:59.650838 27270 tablet_bootstrap.cc:492] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:09:59.651055 27270 ts_tablet_manager.cc:1367] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 20:09:59.655175 27270 raft_consensus.cc:348] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.655517 27270 raft_consensus.cc:374] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.655689 27270 raft_consensus.cc:729] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.655988 27270 consensus_queue.cc:260] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.657831 27264 raft_consensus.cc:482] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:59.657965 27264 raft_consensus.cc:504] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.658674 27264 leader_election.cc:288] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:59.659319 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb552d92b68a40b4b35c995c98291c91" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
W20240501 20:09:59.659983 24046 leader_election.cc:341] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:59.660516 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb552d92b68a40b4b35c995c98291c91" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:09:59.660710 23846 raft_consensus.cc:2454] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:09:59.661144 24046 leader_election.cc:302] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:09:59.661612 27264 raft_consensus.cc:2781] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:59.661715 27264 raft_consensus.cc:482] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:59.661787 27264 raft_consensus.cc:3037] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.665478 27264 raft_consensus.cc:504] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.667862 27156 raft_consensus.cc:482] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:59.668027 27259 tablet_bootstrap.cc:492] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:59.668023 27156 raft_consensus.cc:504] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.668171 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb552d92b68a40b4b35c995c98291c91" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:59.668715 27156 leader_election.cc:288] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
W20240501 20:09:59.668723 24046 leader_election.cc:341] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:59.669456 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb552d92b68a40b4b35c995c98291c91" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:09:59.669651 23846 raft_consensus.cc:3037] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.669611 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7eb4c409a2945f0b39bebd37fec4ea7" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:09:59.669816 21357 raft_consensus.cc:2454] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:09:59.670346 20901 leader_election.cc:302] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:59.671006 27156 raft_consensus.cc:2781] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:59.671097 27156 raft_consensus.cc:482] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:59.671156 27156 raft_consensus.cc:3037] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.672510 23846 raft_consensus.cc:2454] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:09:59.672891 24046 leader_election.cc:302] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:09:59.673197 27030 raft_consensus.cc:2781] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:59.673826 27156 raft_consensus.cc:504] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.675151 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7eb4c409a2945f0b39bebd37fec4ea7" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:09:59.675376 21357 raft_consensus.cc:3037] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.675372 27264 leader_election.cc:288] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:09:59.678110 21357 raft_consensus.cc:2454] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:59.678867 20901 leader_election.cc:302] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:09:59.679209 27253 raft_consensus.cc:2781] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:59.679335 27253 raft_consensus.cc:686] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:09:59.679631 27253 consensus_queue.cc:237] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.681259 27270 ts_tablet_manager.cc:1398] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20240501 20:09:59.681690 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7eb4c409a2945f0b39bebd37fec4ea7" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
W20240501 20:09:59.682380 20903 leader_election.cc:341] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:59.668186 27259 ts_tablet_manager.cc:1367] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.002s
I20240501 20:09:59.683331 27156 leader_election.cc:288] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:09:59.683422 27259 raft_consensus.cc:348] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.683609 27259 raft_consensus.cc:374] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.683657 27259 raft_consensus.cc:729] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.685039 27259 consensus_queue.cc:260] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.687423 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7eb4c409a2945f0b39bebd37fec4ea7" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:09:59.687572 27259 ts_tablet_manager.cc:1398] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 20:09:59.687625 21228 raft_consensus.cc:3037] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.687891 27259 tablet_bootstrap.cc:492] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:59.689081 27030 raft_consensus.cc:686] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:09:59.689556 27030 consensus_queue.cc:237] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.690277 21228 raft_consensus.cc:2454] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:09:59.692350 27259 tablet_bootstrap.cc:654] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.699139 27259 tablet_bootstrap.cc:492] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:59.699290 27259 ts_tablet_manager.cc:1367] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 20:09:59.701488 27259 raft_consensus.cc:348] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.701735 27259 raft_consensus.cc:374] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.701783 27259 raft_consensus.cc:729] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.706621 20819 catalog_manager.cc:5484] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:59.706545 20823 catalog_manager.cc:5484] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:09:59.708567 27259 consensus_queue.cc:260] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:09:59.709708 27259 ts_tablet_manager.cc:1398] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:09:59.710105 27259 tablet_bootstrap.cc:492] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:09:59.712982 27259 tablet_bootstrap.cc:654] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.719398 27259 tablet_bootstrap.cc:492] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:09:59.719570 27259 ts_tablet_manager.cc:1367] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240501 20:09:59.720297 27259 raft_consensus.cc:348] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.720495 27259 raft_consensus.cc:374] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.720549 27259 raft_consensus.cc:729] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.720760 27259 consensus_queue.cc:260] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.721269 27259 ts_tablet_manager.cc:1398] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:59.793068 27254 raft_consensus.cc:482] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:59.793234 27254 raft_consensus.cc:504] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.793934 27254 leader_election.cc:288] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:59.794324 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ccc5b4ec949f1aa0652e43a3b9e5c" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:09:59.794555 21098 raft_consensus.cc:2454] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:59.794554 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ccc5b4ec949f1aa0652e43a3b9e5c" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:09:59.794764 24109 raft_consensus.cc:2454] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:09:59.795188 21163 leader_election.cc:302] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:09:59.795529 27254 raft_consensus.cc:2781] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:59.795626 27254 raft_consensus.cc:482] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:59.795684 27254 raft_consensus.cc:3037] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.798130 27254 raft_consensus.cc:504] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.798830 27254 leader_election.cc:288] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:09:59.799078 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ccc5b4ec949f1aa0652e43a3b9e5c" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:09:59.799269 21098 raft_consensus.cc:3037] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.800050 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ccc5b4ec949f1aa0652e43a3b9e5c" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:09:59.800298 24109 raft_consensus.cc:3037] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.801527 21098 raft_consensus.cc:2454] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:59.804410 21161 leader_election.cc:302] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:09:59.804713 27164 raft_consensus.cc:2781] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:59.804883 27164 raft_consensus.cc:686] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:09:59.805213 27164 consensus_queue.cc:237] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:09:59.805485 24109 raft_consensus.cc:2454] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:09:59.809494 20826 catalog_manager.cc:5484] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:09:59.816051 21228 consensus_queue.cc:237] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:09:59.817978 24109 raft_consensus.cc:1261] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:09:59.818070 21357 raft_consensus.cc:1261] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:09:59.818557 27164 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:59.818853 27254 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:59.821327 27164 raft_consensus.cc:2932] T ed53558d93214d34aedca8505870ec61 P 4af096f5a6a9440a94f6682b399efe20 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:59.822057 24109 raft_consensus.cc:2932] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:59.825830 21357 raft_consensus.cc:2932] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:59.828689 20826 catalog_manager.cc:5484] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 2 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 2 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:09:59.836809 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet ed53558d93214d34aedca8505870ec61 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57076
I20240501 20:09:59.837976 27307 tablet_replica.cc:330] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:09:59.838351 27307 raft_consensus.cc:2229] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:59.838704 27307 raft_consensus.cc:2258] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.841157 27307 ts_tablet_manager.cc:1875] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:59.846743 27307 ts_tablet_manager.cc:1888] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 20:09:59.846890 27307 log.cc:1196] T ed53558d93214d34aedca8505870ec61 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/ed53558d93214d34aedca8505870ec61
I20240501 20:09:59.848114 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet ed53558d93214d34aedca8505870ec61 (table rebalancer_extra_table_6 [id=74165ecf88a44e599659026739b1c993]) successfully deleted
I20240501 20:09:59.849572 20819 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:59.873795 23846 raft_consensus.cc:1261] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:59.874567 27030 consensus_queue.cc:1035] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:59.882185 21228 raft_consensus.cc:3037] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.886123 21228 raft_consensus.cc:1261] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:59.886942 27030 consensus_queue.cc:1035] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:59.896330 20819 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet eb552d92b68a40b4b35c995c98291c91 (table rebalancer_dynamic_table [id=200a5ec20e86403bab80c66e8dd8da14]). Expected version 1 got 0
I20240501 20:09:59.900753 27320 tablet.cc:1722] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:59.901322 20826 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet eb552d92b68a40b4b35c995c98291c91 (table rebalancer_dynamic_table [id=200a5ec20e86403bab80c66e8dd8da14]). Expected version 1 got 0
I20240501 20:09:59.901804 27320 tablet.cc:1620] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:09:59.910995 27322 tablet.cc:1722] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:59.914428 27321 tablet.cc:1722] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:59.915043 20826 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet eb552d92b68a40b4b35c995c98291c91 (table rebalancer_dynamic_table [id=200a5ec20e86403bab80c66e8dd8da14]). Expected version 1 got 0
I20240501 20:09:59.915277 27321 tablet.cc:1620] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:09:59.917172 20819 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet eb552d92b68a40b4b35c995c98291c91 (table rebalancer_dynamic_table [id=200a5ec20e86403bab80c66e8dd8da14]). Expected version 1 got 0
I20240501 20:09:59.917953 27322 tablet.cc:1620] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:09:59.935611 20819 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=200a5ec20e86403bab80c66e8dd8da14] alter complete (version 1)
I20240501 20:10:00.034320 21228 raft_consensus.cc:3037] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.037482 27156 consensus_queue.cc:1035] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.045989 20819 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:00.052138 27156 consensus_queue.cc:1035] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.057873 27320 tablet.cc:1722] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:00.058487 27320 tablet.cc:1620] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:10:00.060590 27321 tablet.cc:1722] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:00.060950 27321 tablet.cc:1620] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:10:00.065449 27322 tablet.cc:1722] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:00.065919 27322 tablet.cc:1620] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:10:00.082573 20823 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=200a5ec20e86403bab80c66e8dd8da14] alter complete (version 2)
I20240501 20:10:00.086445 27156 consensus_queue.cc:1035] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.147311 27055 consensus_queue.cc:1035] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.186616 20819 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:00.200434 27320 tablet.cc:1722] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:00.202469 27321 tablet.cc:1722] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:00.211983 20819 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=200a5ec20e86403bab80c66e8dd8da14] alter complete (version 3)
I20240501 20:10:00.215276 27322 tablet.cc:1722] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:00.316418 20820 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:00.316825 20820 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:00.322108 20820 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet eb552d92b68a40b4b35c995c98291c91
I20240501 20:10:00.323050 21206 tablet_service.cc:1514] Processing DeleteTablet for tablet eb552d92b68a40b4b35c995c98291c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:10:00.323415 23824 tablet_service.cc:1514] Processing DeleteTablet for tablet eb552d92b68a40b4b35c995c98291c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:10:00.323796 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet eb552d92b68a40b4b35c995c98291c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:00.323832 27307 tablet_replica.cc:330] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:10:00.324107 27307 raft_consensus.cc:2229] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.324393 27307 raft_consensus.cc:2258] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.326000 27307 ts_tablet_manager.cc:1875] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.326196 27164 consensus_queue.cc:1035] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.336664 27307 ts_tablet_manager.cc:1888] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:00.336828 27307 log.cc:1196] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/eb552d92b68a40b4b35c995c98291c91
I20240501 20:10:00.337409 27307 ts_tablet_manager.cc:1909] T eb552d92b68a40b4b35c995c98291c91 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:10:00.338642 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet eb552d92b68a40b4b35c995c98291c91 (table rebalancer_dynamic_table_new_name [id=200a5ec20e86403bab80c66e8dd8da14]) successfully deleted
I20240501 20:10:00.349368 27373 tablet_replica.cc:330] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:00.350739 27373 raft_consensus.cc:2229] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:00.351603 27373 raft_consensus.cc:2258] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.355996 27373 ts_tablet_manager.cc:1875] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.365615 27372 tablet_replica.cc:330] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:10:00.365826 27372 raft_consensus.cc:2229] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.366025 27372 raft_consensus.cc:2258] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.367149 20826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:00.370005 27372 ts_tablet_manager.cc:1875] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.371234 27164 consensus_queue.cc:1035] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.371874 27373 ts_tablet_manager.cc:1888] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:00.372071 27373 log.cc:1196] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/eb552d92b68a40b4b35c995c98291c91
I20240501 20:10:00.373031 27373 ts_tablet_manager.cc:1909] T eb552d92b68a40b4b35c995c98291c91 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:10:00.374138 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet eb552d92b68a40b4b35c995c98291c91 (table rebalancer_dynamic_table_new_name [id=200a5ec20e86403bab80c66e8dd8da14]) successfully deleted
I20240501 20:10:00.380311 27372 ts_tablet_manager.cc:1888] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:00.380535 27372 log.cc:1196] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb552d92b68a40b4b35c995c98291c91
I20240501 20:10:00.381018 27372 ts_tablet_manager.cc:1909] T eb552d92b68a40b4b35c995c98291c91 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:10:00.382694 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet eb552d92b68a40b4b35c995c98291c91 (table rebalancer_dynamic_table_new_name [id=200a5ec20e86403bab80c66e8dd8da14]) successfully deleted
I20240501 20:10:00.385159 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 46677b6d96bf45d98d061e572a85ebdc (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.385370 23689 tablet_service.cc:1467] Processing CreateTablet for tablet 46677b6d96bf45d98d061e572a85ebdc (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.385573 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46677b6d96bf45d98d061e572a85ebdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.385829 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46677b6d96bf45d98d061e572a85ebdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.387981 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 46677b6d96bf45d98d061e572a85ebdc (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.388449 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46677b6d96bf45d98d061e572a85ebdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.388536 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 5678961411804cd08599d10d15a2aa7f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.388953 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5678961411804cd08599d10d15a2aa7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.391428 21336 tablet_service.cc:1467] Processing CreateTablet for tablet fc7ec7aaaaf14c6698dfad70c1ca1e32 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.391610 27378 tablet_bootstrap.cc:492] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:00.391877 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7ec7aaaaf14c6698dfad70c1ca1e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.391886 23824 tablet_service.cc:1467] Processing CreateTablet for tablet fc7ec7aaaaf14c6698dfad70c1ca1e32 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.392305 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7ec7aaaaf14c6698dfad70c1ca1e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.393409 27378 tablet_bootstrap.cc:654] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.395529 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 5678961411804cd08599d10d15a2aa7f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.395969 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5678961411804cd08599d10d15a2aa7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.395967 20947 tablet_service.cc:1467] Processing CreateTablet for tablet fc7ec7aaaaf14c6698dfad70c1ca1e32 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.396348 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7ec7aaaaf14c6698dfad70c1ca1e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.404714 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 5678961411804cd08599d10d15a2aa7f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.405205 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5678961411804cd08599d10d15a2aa7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.417464 27381 tablet_bootstrap.cc:492] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:00.417800 27382 tablet_bootstrap.cc:492] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:00.419144 27381 tablet_bootstrap.cc:654] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.419145 27382 tablet_bootstrap.cc:654] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.422196 27378 tablet_bootstrap.cc:492] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:00.422497 27378 ts_tablet_manager.cc:1367] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.010s
I20240501 20:10:00.427274 27381 tablet_bootstrap.cc:492] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:10:00.427572 27381 ts_tablet_manager.cc:1367] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 20:10:00.428375 27381 raft_consensus.cc:348] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:00.428668 27381 raft_consensus.cc:374] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.428781 27381 raft_consensus.cc:729] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.429144 27381 consensus_queue.cc:260] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:00.429751 27381 ts_tablet_manager.cc:1398] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:00.430924 27378 raft_consensus.cc:348] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.432065 27378 raft_consensus.cc:374] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.432312 27378 raft_consensus.cc:729] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.432780 27378 consensus_queue.cc:260] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.436048 27378 ts_tablet_manager.cc:1398] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 20:10:00.436177 27383 tablet_bootstrap.cc:492] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:00.440598 27378 tablet_bootstrap.cc:492] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:00.443225 27378 tablet_bootstrap.cc:654] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.446832 27387 tablet_bootstrap.cc:492] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:00.447150 27383 tablet_bootstrap.cc:654] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.448930 27378 tablet_bootstrap.cc:492] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:00.449240 27378 ts_tablet_manager.cc:1367] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 20:10:00.450431 27378 raft_consensus.cc:348] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:00.450731 27378 raft_consensus.cc:374] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.450836 27378 raft_consensus.cc:729] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.450865 27387 tablet_bootstrap.cc:654] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.451160 27378 consensus_queue.cc:260] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:00.451874 27378 ts_tablet_manager.cc:1398] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:00.452910 27382 tablet_bootstrap.cc:492] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:10:00.453091 27382 ts_tablet_manager.cc:1367] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.003s
I20240501 20:10:00.453953 20826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:00.454530 27382 raft_consensus.cc:348] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:00.454792 27382 raft_consensus.cc:374] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.454890 27382 raft_consensus.cc:729] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.455179 27382 consensus_queue.cc:260] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:00.455750 27388 tablet_bootstrap.cc:492] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:00.457608 27388 tablet_bootstrap.cc:654] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.460034 27380 tablet_bootstrap.cc:492] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:00.469393 27382 ts_tablet_manager.cc:1398] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.016s user 0.000s sys 0.001s
I20240501 20:10:00.473670 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbfde0acc4644e5a7054662ec70c3b4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0090222355a747d996e0fca1e610f02b]), partition=
I20240501 20:10:00.474256 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbfde0acc4644e5a7054662ec70c3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.481699 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbfde0acc4644e5a7054662ec70c3b4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0090222355a747d996e0fca1e610f02b]), partition=
I20240501 20:10:00.481894 27382 tablet_bootstrap.cc:492] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:00.482208 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbfde0acc4644e5a7054662ec70c3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.483404 27382 tablet_bootstrap.cc:654] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.489900 27383 tablet_bootstrap.cc:492] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:10:00.490068 27383 ts_tablet_manager.cc:1367] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20240501 20:10:00.490864 27383 raft_consensus.cc:348] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:00.491070 27383 raft_consensus.cc:374] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.491144 27383 raft_consensus.cc:729] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.491417 27383 consensus_queue.cc:260] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:00.492019 27383 ts_tablet_manager.cc:1398] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:00.468473 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbfde0acc4644e5a7054662ec70c3b4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0090222355a747d996e0fca1e610f02b]), partition=
I20240501 20:10:00.492416 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbfde0acc4644e5a7054662ec70c3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.494093 27388 tablet_bootstrap.cc:492] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:10:00.494235 27388 ts_tablet_manager.cc:1367] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.003s
I20240501 20:10:00.496771 27378 tablet_bootstrap.cc:492] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:00.497901 27380 tablet_bootstrap.cc:654] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.498368 27378 tablet_bootstrap.cc:654] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.497587 27388 raft_consensus.cc:348] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:00.500886 27388 raft_consensus.cc:374] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.501014 27388 raft_consensus.cc:729] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.501325 27388 consensus_queue.cc:260] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:00.504770 27388 ts_tablet_manager.cc:1398] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240501 20:10:00.507612 27387 tablet_bootstrap.cc:492] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:00.507890 27387 ts_tablet_manager.cc:1367] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.012s
I20240501 20:10:00.514816 27387 raft_consensus.cc:348] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.516396 27387 raft_consensus.cc:374] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.516722 27387 raft_consensus.cc:729] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.517480 27387 consensus_queue.cc:260] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.525310 26476 raft_consensus.cc:482] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:00.525490 26476 raft_consensus.cc:504] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:00.526304 26476 leader_election.cc:288] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:00.527016 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7ec7aaaaf14c6698dfad70c1ca1e32" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:10:00.527244 20964 raft_consensus.cc:2454] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:10:00.527753 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7ec7aaaaf14c6698dfad70c1ca1e32" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:10:00.527966 21357 raft_consensus.cc:2454] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:10:00.528514 23779 leader_election.cc:302] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:10:00.536072 26476 raft_consensus.cc:2781] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:00.536207 26476 raft_consensus.cc:482] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:00.536314 26476 raft_consensus.cc:3037] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.539768 26476 raft_consensus.cc:504] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:00.541894 27387 ts_tablet_manager.cc:1398] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240501 20:10:00.542116 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7ec7aaaaf14c6698dfad70c1ca1e32" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:10:00.542337 21357 raft_consensus.cc:3037] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.542335 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7ec7aaaaf14c6698dfad70c1ca1e32" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:10:00.542515 20964 raft_consensus.cc:3037] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.545338 20964 raft_consensus.cc:2454] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:10:00.545893 23779 leader_election.cc:302] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:10:00.546260 26720 raft_consensus.cc:2781] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:00.547195 27380 tablet_bootstrap.cc:492] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:10:00.547423 27380 ts_tablet_manager.cc:1367] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.088s user 0.000s sys 0.004s
I20240501 20:10:00.548310 21357 raft_consensus.cc:2454] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:10:00.548379 27380 raft_consensus.cc:348] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:00.548622 27387 tablet_bootstrap.cc:492] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:00.548619 27380 raft_consensus.cc:374] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.548727 27380 raft_consensus.cc:729] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.549149 27380 consensus_queue.cc:260] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:00.549422 26476 leader_election.cc:288] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:00.550038 27380 ts_tablet_manager.cc:1398] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:00.550391 27380 tablet_bootstrap.cc:492] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:00.550558 27387 tablet_bootstrap.cc:654] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.552304 27380 tablet_bootstrap.cc:654] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.559031 27378 tablet_bootstrap.cc:492] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:00.560072 27378 ts_tablet_manager.cc:1367] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.009s
I20240501 20:10:00.561316 27378 raft_consensus.cc:348] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.570163 27378 raft_consensus.cc:374] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.570654 27378 raft_consensus.cc:729] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.575716 26720 raft_consensus.cc:686] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:10:00.576112 26720 consensus_queue.cc:237] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:00.579139 27378 consensus_queue.cc:260] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.580190 27378 ts_tablet_manager.cc:1398] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240501 20:10:00.588632 27382 tablet_bootstrap.cc:492] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:10:00.588866 27382 ts_tablet_manager.cc:1367] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.013s
I20240501 20:10:00.589855 27382 raft_consensus.cc:348] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.590219 27382 raft_consensus.cc:374] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.590329 27382 raft_consensus.cc:729] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.590636 27382 consensus_queue.cc:260] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.591807 27382 ts_tablet_manager.cc:1398] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:00.620013 20820 catalog_manager.cc:5484] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:00.632577 21206 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:00.657366 21336 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 20:10:00.666779 27386 raft_consensus.cc:482] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:00.666970 27386 raft_consensus.cc:504] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.667795 27386 leader_election.cc:288] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:00.668605 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46677b6d96bf45d98d061e572a85ebdc" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:10:00.668833 23977 raft_consensus.cc:2454] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:10:00.671041 23644 leader_election.cc:302] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:10:00.671239 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46677b6d96bf45d98d061e572a85ebdc" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:10:00.671391 27386 raft_consensus.cc:2781] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:00.671458 20964 raft_consensus.cc:2454] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:10:00.671490 27386 raft_consensus.cc:482] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:00.671698 27386 raft_consensus.cc:3037] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.676048 27386 raft_consensus.cc:504] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.676939 27386 leader_election.cc:288] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:00.677693 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46677b6d96bf45d98d061e572a85ebdc" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:10:00.677909 23977 raft_consensus.cc:3037] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.683771 23977 raft_consensus.cc:2454] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:10:00.687764 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46677b6d96bf45d98d061e572a85ebdc" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:10:00.688002 20964 raft_consensus.cc:3037] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.688246 23644 leader_election.cc:302] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:10:00.688656 27386 raft_consensus.cc:2781] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:00.689167 27386 raft_consensus.cc:686] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:10:00.689580 27386 consensus_queue.cc:237] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.690920 20964 raft_consensus.cc:2454] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:10:00.695561 27387 tablet_bootstrap.cc:492] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:00.697741 23691 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:00.698755 27387 ts_tablet_manager.cc:1367] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.149s user 0.003s sys 0.017s
I20240501 20:10:00.700490 27380 tablet_bootstrap.cc:492] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:10:00.701092 24089 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:00.706634 27380 ts_tablet_manager.cc:1367] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.156s user 0.004s sys 0.000s
I20240501 20:10:00.707571 27380 raft_consensus.cc:348] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.707803 27380 raft_consensus.cc:374] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.707868 27380 raft_consensus.cc:729] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.709820 20820 catalog_manager.cc:5484] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:00.714756 27387 raft_consensus.cc:348] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.715523 27387 raft_consensus.cc:374] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.715692 27387 raft_consensus.cc:729] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.716022 27387 consensus_queue.cc:260] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.717944 27387 ts_tablet_manager.cc:1398] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240501 20:10:00.736529 27375 raft_consensus.cc:482] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:00.736739 27375 raft_consensus.cc:504] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:00.737491 27375 leader_election.cc:288] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:00.738032 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5678961411804cd08599d10d15a2aa7f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:00.738299 23712 raft_consensus.cc:2454] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:10:00.738382 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5678961411804cd08599d10d15a2aa7f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:10:00.738633 24109 raft_consensus.cc:2454] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:10:00.738804 21033 leader_election.cc:302] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:00.739360 27375 raft_consensus.cc:2781] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:00.739461 27375 raft_consensus.cc:482] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:00.739574 27375 raft_consensus.cc:3037] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.742403 27375 raft_consensus.cc:504] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:00.743063 27375 leader_election.cc:288] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:00.743717 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5678961411804cd08599d10d15a2aa7f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:10:00.743945 23712 raft_consensus.cc:3037] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.744021 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5678961411804cd08599d10d15a2aa7f" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:10:00.744211 24110 raft_consensus.cc:3037] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.746555 23712 raft_consensus.cc:2454] T 5678961411804cd08599d10d15a2aa7f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:10:00.747112 24110 raft_consensus.cc:2454] T 5678961411804cd08599d10d15a2aa7f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:10:00.747485 21033 leader_election.cc:302] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:00.747901 26745 raft_consensus.cc:2781] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:00.748023 26745 raft_consensus.cc:686] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:10:00.748247 26745 consensus_queue.cc:237] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:00.764559 27380 consensus_queue.cc:260] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.769771 23957 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:00.772429 27380 ts_tablet_manager.cc:1398] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.066s user 0.000s sys 0.002s
W20240501 20:10:00.816345 21033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 62795.9us
I20240501 20:10:00.818944 23824 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:00.822736 21076 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:00.823645 20820 catalog_manager.cc:5484] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:00.832664 20945 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:10:00.959627 27285 raft_consensus.cc:482] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:00.959817 27285 raft_consensus.cc:504] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.960552 27285 leader_election.cc:288] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:00.961491 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbfde0acc4644e5a7054662ec70c3b4" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:10:00.961771 23977 raft_consensus.cc:2454] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:10:00.962929 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbfde0acc4644e5a7054662ec70c3b4" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:00.963174 23712 raft_consensus.cc:2454] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:10:00.964689 24043 leader_election.cc:302] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:10:00.965246 27030 raft_consensus.cc:2781] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:00.965370 27030 raft_consensus.cc:482] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:00.965459 27030 raft_consensus.cc:3037] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.968253 27030 raft_consensus.cc:504] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:00.969272 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbfde0acc4644e5a7054662ec70c3b4" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:10:00.969496 23977 raft_consensus.cc:3037] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.972306 23977 raft_consensus.cc:2454] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:10:00.973033 24043 leader_election.cc:302] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:10:00.973295 27285 raft_consensus.cc:2781] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:00.973637 27030 leader_election.cc:288] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:00.974099 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbfde0acc4644e5a7054662ec70c3b4" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:10:00.974344 23712 raft_consensus.cc:3037] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.977054 23712 raft_consensus.cc:2454] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:10:01.004045 27285 raft_consensus.cc:686] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:10:01.004608 27285 consensus_queue.cc:237] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.024839 20820 catalog_manager.cc:5484] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:01.075709 26720 consensus_queue.cc:1035] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.110879 27021 consensus_queue.cc:1035] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.133685 26720 consensus_queue.cc:1035] T fc7ec7aaaaf14c6698dfad70c1ca1e32 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.152829 27021 consensus_queue.cc:1035] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.178256 27425 consensus_queue.cc:1035] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.204130 23976 consensus_queue.cc:237] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:01.221331 21357 raft_consensus.cc:1261] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:01.221985 20965 raft_consensus.cc:1261] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:01.222236 27375 consensus_queue.cc:1035] T 5678961411804cd08599d10d15a2aa7f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.222970 27114 consensus_queue.cc:1035] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:01.223368 27327 consensus_queue.cc:1035] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:01.228894 27114 raft_consensus.cc:2932] T a9fd7894fc5745ae9588437cd5bd0265 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:01.235198 20826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:01.234961 21357 raft_consensus.cc:2932] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:01.245187 23710 raft_consensus.cc:1261] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:01.246124 23977 raft_consensus.cc:1261] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:01.246842 27426 consensus_queue.cc:1035] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.247902 27426 consensus_queue.cc:1035] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.251353 20965 raft_consensus.cc:2932] T a9fd7894fc5745ae9588437cd5bd0265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:01.253109 20826 catalog_manager.cc:5484] T a9fd7894fc5745ae9588437cd5bd0265 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:01.261117 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fd7894fc5745ae9588437cd5bd0265 with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:01.262269 20821 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 7bbfde0acc4644e5a7054662ec70c3b4 (table rebalancer_dynamic_table [id=0090222355a747d996e0fca1e610f02b]). Expected version 1 got 0
I20240501 20:10:01.270124 27468 tablet.cc:1722] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:01.270916 27468 tablet.cc:1620] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:10:01.280563 27469 tablet.cc:1722] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:01.283785 27469 tablet.cc:1620] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:10:01.295994 20826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=0090222355a747d996e0fca1e610f02b] alter complete (version 1)
I20240501 20:10:01.302953 20820 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 7bbfde0acc4644e5a7054662ec70c3b4 (table rebalancer_dynamic_table [id=0090222355a747d996e0fca1e610f02b]). Expected version 1 got 0
I20240501 20:10:01.313009 27458 mvcc.cc:205] Tried to move back new op lower bound from 7022977848280940544 to 7022977847321559040. Current Snapshot: MvccSnapshot[applied={T|T < 7022977848280940544}]
I20240501 20:10:01.318722 27465 tablet_replica.cc:330] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:01.319243 27467 tablet.cc:1722] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:01.319779 27467 tablet.cc:1620] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:10:01.319763 27465 raft_consensus.cc:2229] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.320103 27465 raft_consensus.cc:2258] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.321025 27465 ts_tablet_manager.cc:1875] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:10:01.321508 23910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 72698.5us
I20240501 20:10:01.329913 27465 ts_tablet_manager.cc:1888] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:01.330129 27465 log.cc:1196] T a9fd7894fc5745ae9588437cd5bd0265 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/a9fd7894fc5745ae9588437cd5bd0265
I20240501 20:10:01.332288 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet a9fd7894fc5745ae9588437cd5bd0265 (table rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]) successfully deleted
I20240501 20:10:01.356635 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:01.374434 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 6d11505bdd2243949a9776989410e534 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.374876 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d11505bdd2243949a9776989410e534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.378050 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 6d11505bdd2243949a9776989410e534 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.378474 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d11505bdd2243949a9776989410e534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.379106 21336 tablet_service.cc:1467] Processing CreateTablet for tablet a7498fc5cf5144d58712eb2c8f188c5e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.379698 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7498fc5cf5144d58712eb2c8f188c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.382826 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 6d11505bdd2243949a9776989410e534 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.383358 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d11505bdd2243949a9776989410e534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.387351 23824 tablet_service.cc:1467] Processing CreateTablet for tablet a7498fc5cf5144d58712eb2c8f188c5e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.387817 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7498fc5cf5144d58712eb2c8f188c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.393942 21077 tablet_service.cc:1467] Processing CreateTablet for tablet a7498fc5cf5144d58712eb2c8f188c5e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.394546 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7498fc5cf5144d58712eb2c8f188c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.411150 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecad173c71744eca8ca537ad152f95f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.411763 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecad173c71744eca8ca537ad152f95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.413605 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:01.423700 27483 tablet_bootstrap.cc:492] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:01.426043 21337 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecad173c71744eca8ca537ad152f95f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.426440 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecad173c71744eca8ca537ad152f95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.427987 27485 tablet_bootstrap.cc:492] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:01.430148 27482 tablet_bootstrap.cc:492] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:01.430153 27467 tablet.cc:1722] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:01.430701 27467 tablet.cc:1620] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: MemRowSet was empty: no flush needed.
I20240501 20:10:01.430735 27468 tablet.cc:1722] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:01.431152 27468 tablet.cc:1620] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:10:01.432636 27485 tablet_bootstrap.cc:654] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.432896 27483 tablet_bootstrap.cc:654] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.436441 27469 tablet.cc:1722] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:01.436887 27469 tablet.cc:1620] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:10:01.440232 27484 tablet_bootstrap.cc:492] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:01.443675 27484 tablet_bootstrap.cc:654] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.452500 27482 tablet_bootstrap.cc:654] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.464885 27484 tablet_bootstrap.cc:492] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:10:01.464920 27482 tablet_bootstrap.cc:492] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:10:01.465081 27482 ts_tablet_manager.cc:1367] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.000s
I20240501 20:10:01.466068 27484 ts_tablet_manager.cc:1367] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.005s
I20240501 20:10:01.467270 27484 raft_consensus.cc:348] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.467412 27482 raft_consensus.cc:348] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:01.468183 27482 raft_consensus.cc:374] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.468580 27482 raft_consensus.cc:729] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.469089 27482 consensus_queue.cc:260] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:01.469256 20820 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=0090222355a747d996e0fca1e610f02b] alter complete (version 2)
I20240501 20:10:01.470185 27485 tablet_bootstrap.cc:492] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:10:01.470702 27485 ts_tablet_manager.cc:1367] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.005s
I20240501 20:10:01.471628 27484 raft_consensus.cc:374] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.471781 27484 raft_consensus.cc:729] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.472126 27484 consensus_queue.cc:260] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.472378 27482 ts_tablet_manager.cc:1398] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:01.472654 27485 raft_consensus.cc:348] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.475656 27485 raft_consensus.cc:374] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.476145 27485 raft_consensus.cc:729] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.479003 27484 ts_tablet_manager.cc:1398] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240501 20:10:01.483125 27483 tablet_bootstrap.cc:492] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:01.483522 27483 ts_tablet_manager.cc:1367] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.000s
I20240501 20:10:01.484959 27483 raft_consensus.cc:348] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.485559 27483 raft_consensus.cc:374] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.485780 27483 raft_consensus.cc:729] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.479609 27485 consensus_queue.cc:260] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.486474 27483 consensus_queue.cc:260] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.500701 27483 ts_tablet_manager.cc:1398] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240501 20:10:01.493842 27485 ts_tablet_manager.cc:1398] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240501 20:10:01.509088 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecad173c71744eca8ca537ad152f95f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.509611 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecad173c71744eca8ca537ad152f95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.520363 27484 tablet_bootstrap.cc:492] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:01.520617 27495 tablet_bootstrap.cc:492] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:01.522228 27484 tablet_bootstrap.cc:654] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.522312 27495 tablet_bootstrap.cc:654] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.539060 27485 tablet_bootstrap.cc:492] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:01.542266 27484 tablet_bootstrap.cc:492] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:10:01.542496 27484 ts_tablet_manager.cc:1367] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
I20240501 20:10:01.543329 27484 raft_consensus.cc:348] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:01.543623 27484 raft_consensus.cc:374] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.543720 27484 raft_consensus.cc:729] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.544019 27484 consensus_queue.cc:260] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:01.549018 27495 tablet_bootstrap.cc:492] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:10:01.549487 27495 ts_tablet_manager.cc:1367] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240501 20:10:01.550427 27495 raft_consensus.cc:348] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:01.550709 27495 raft_consensus.cc:374] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.550865 27495 raft_consensus.cc:729] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.551136 27495 consensus_queue.cc:260] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:01.552464 27484 ts_tablet_manager.cc:1398] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 20:10:01.553464 27495 ts_tablet_manager.cc:1398] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:01.552063 27485 tablet_bootstrap.cc:654] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.568343 27485 tablet_bootstrap.cc:492] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:10:01.568509 27485 ts_tablet_manager.cc:1367] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20240501 20:10:01.569428 27485 raft_consensus.cc:348] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:01.569648 27485 raft_consensus.cc:374] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.569720 27485 raft_consensus.cc:729] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.569993 27485 consensus_queue.cc:260] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:01.570600 27485 ts_tablet_manager.cc:1398] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:01.571105 27481 tablet_bootstrap.cc:492] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:01.573879 27481 tablet_bootstrap.cc:654] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.577415 27407 raft_consensus.cc:482] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:01.577648 27407 raft_consensus.cc:504] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:01.578613 27407 leader_election.cc:288] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:10:01.579463 21355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7498fc5cf5144d58712eb2c8f188c5e" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
W20240501 20:10:01.580523 23780 leader_election.cc:341] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:01.581550 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7498fc5cf5144d58712eb2c8f188c5e" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:10:01.581815 21098 raft_consensus.cc:2454] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:10:01.583015 23779 leader_election.cc:302] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters: 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:10:01.583799 27407 raft_consensus.cc:2781] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:01.583922 27407 raft_consensus.cc:482] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:01.584007 27407 raft_consensus.cc:3037] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.588757 27481 tablet_bootstrap.cc:492] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:10:01.588948 27481 ts_tablet_manager.cc:1367] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 20:10:01.589646 27481 raft_consensus.cc:348] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:01.589885 27481 raft_consensus.cc:374] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.589982 27481 raft_consensus.cc:729] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.590237 27481 consensus_queue.cc:260] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:01.590780 27481 ts_tablet_manager.cc:1398] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:01.591045 27481 tablet_bootstrap.cc:492] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:01.592618 27481 tablet_bootstrap.cc:654] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.611757 20821 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:01.613351 27407 raft_consensus.cc:504] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:01.614025 27481 tablet_bootstrap.cc:492] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:10:01.614328 27481 ts_tablet_manager.cc:1367] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20240501 20:10:01.614602 21355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7498fc5cf5144d58712eb2c8f188c5e" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:10:01.614893 21355 raft_consensus.cc:3037] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.617024 27481 raft_consensus.cc:348] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:01.617172 27481 raft_consensus.cc:374] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.617244 27481 raft_consensus.cc:729] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.617450 27481 consensus_queue.cc:260] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:01.618007 27481 ts_tablet_manager.cc:1398] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 20:10:01.621302 27467 tablet.cc:1722] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:01.621721 27468 tablet.cc:1722] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:01.631278 27469 tablet.cc:1722] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:01.638433 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=0090222355a747d996e0fca1e610f02b] alter complete (version 3)
I20240501 20:10:01.650182 21355 raft_consensus.cc:2454] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:10:01.650956 23780 leader_election.cc:302] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:10:01.651520 26720 raft_consensus.cc:2781] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:01.677867 27407 leader_election.cc:288] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:10:01.680418 26745 raft_consensus.cc:482] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:01.680646 26745 raft_consensus.cc:504] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.682315 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d11505bdd2243949a9776989410e534" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:10:01.682523 20965 raft_consensus.cc:2454] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:10:01.683020 21031 leader_election.cc:302] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:01.683725 27425 raft_consensus.cc:2781] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:01.683840 27425 raft_consensus.cc:482] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:01.683946 27425 raft_consensus.cc:3037] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.687197 27425 raft_consensus.cc:504] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.688491 26745 leader_election.cc:288] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:01.688462 26720 raft_consensus.cc:686] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:10:01.688767 26720 consensus_queue.cc:237] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:01.689481 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d11505bdd2243949a9776989410e534" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:10:01.689661 20965 raft_consensus.cc:3037] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.689795 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d11505bdd2243949a9776989410e534" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:01.690100 23712 raft_consensus.cc:2454] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:10:01.693478 20965 raft_consensus.cc:2454] T 6d11505bdd2243949a9776989410e534 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:10:01.694612 21031 leader_election.cc:302] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:01.695488 26745 raft_consensus.cc:2781] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:01.695989 26745 raft_consensus.cc:686] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:10:01.696339 26745 consensus_queue.cc:237] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:01.697443 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d11505bdd2243949a9776989410e534" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:10:01.697755 23712 raft_consensus.cc:3037] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.706228 27425 leader_election.cc:288] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:01.707283 20826 catalog_manager.cc:5484] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:01.711086 23712 raft_consensus.cc:2454] T 6d11505bdd2243949a9776989410e534 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:10:01.720784 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7498fc5cf5144d58712eb2c8f188c5e" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:10:01.721074 21098 raft_consensus.cc:3037] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.725281 21098 raft_consensus.cc:2454] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:10:01.750188 20826 catalog_manager.cc:5484] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:01.759598 20826 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:01.759811 20826 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:01.763401 20826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 7bbfde0acc4644e5a7054662ec70c3b4
I20240501 20:10:01.764240 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbfde0acc4644e5a7054662ec70c3b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:01.764338 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbfde0acc4644e5a7054662ec70c3b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:01.764616 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbfde0acc4644e5a7054662ec70c3b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:01.764943 27465 tablet_replica.cc:330] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:01.765172 27465 raft_consensus.cc:2229] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.765411 27465 raft_consensus.cc:2258] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.765662 27514 tablet_replica.cc:330] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:01.766467 27465 ts_tablet_manager.cc:1875] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.766964 27514 raft_consensus.cc:2229] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.767256 27514 raft_consensus.cc:2258] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.773206 27514 ts_tablet_manager.cc:1875] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.773598 27465 ts_tablet_manager.cc:1888] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:01.773773 27465 log.cc:1196] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/7bbfde0acc4644e5a7054662ec70c3b4
I20240501 20:10:01.774289 27465 ts_tablet_manager.cc:1909] T 7bbfde0acc4644e5a7054662ec70c3b4 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:10:01.775331 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 7bbfde0acc4644e5a7054662ec70c3b4 (table rebalancer_dynamic_table_new_name [id=0090222355a747d996e0fca1e610f02b]) successfully deleted
I20240501 20:10:01.783375 27515 tablet_replica.cc:330] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:01.784302 27515 raft_consensus.cc:2229] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:01.784735 27515 raft_consensus.cc:2258] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.787189 27515 ts_tablet_manager.cc:1875] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.788655 27514 ts_tablet_manager.cc:1888] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:01.788833 27514 log.cc:1196] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/7bbfde0acc4644e5a7054662ec70c3b4
I20240501 20:10:01.790942 27514 ts_tablet_manager.cc:1909] T 7bbfde0acc4644e5a7054662ec70c3b4 P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:10:01.791244 27252 raft_consensus.cc:482] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:01.791450 27252 raft_consensus.cc:504] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:01.792420 27252 leader_election.cc:288] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:01.792816 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecad173c71744eca8ca537ad152f95f" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:10:01.793012 20965 raft_consensus.cc:2454] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:10:01.793005 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 7bbfde0acc4644e5a7054662ec70c3b4 (table rebalancer_dynamic_table_new_name [id=0090222355a747d996e0fca1e610f02b]) successfully deleted
I20240501 20:10:01.793490 21291 leader_election.cc:302] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:10:01.793870 27252 raft_consensus.cc:2781] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:01.793937 27252 raft_consensus.cc:482] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:01.793988 27252 raft_consensus.cc:3037] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.794122 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecad173c71744eca8ca537ad152f95f" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:10:01.794430 24109 raft_consensus.cc:2454] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:10:01.800289 27252 raft_consensus.cc:504] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:01.801328 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecad173c71744eca8ca537ad152f95f" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:10:01.801523 20965 raft_consensus.cc:3037] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.804361 20965 raft_consensus.cc:2454] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:10:01.805286 21291 leader_election.cc:302] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:10:01.805635 27345 raft_consensus.cc:2781] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:01.805744 27345 raft_consensus.cc:686] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:10:01.806035 27345 consensus_queue.cc:237] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:01.807763 27252 leader_election.cc:288] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:01.808185 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecad173c71744eca8ca537ad152f95f" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:10:01.809710 24109 raft_consensus.cc:3037] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.814160 24109 raft_consensus.cc:2454] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:10:01.819440 20818 catalog_manager.cc:5484] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:01.820595 27515 ts_tablet_manager.cc:1888] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:01.820715 27515 log.cc:1196] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/7bbfde0acc4644e5a7054662ec70c3b4
I20240501 20:10:01.821199 27515 ts_tablet_manager.cc:1909] T 7bbfde0acc4644e5a7054662ec70c3b4 P 8559229e044b4e1ea661b881b3f9af50: Deleting consensus metadata
I20240501 20:10:01.822356 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 7bbfde0acc4644e5a7054662ec70c3b4 (table rebalancer_dynamic_table_new_name [id=0090222355a747d996e0fca1e610f02b]) successfully deleted
I20240501 20:10:01.867899 20820 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:01.878335 23824 tablet_service.cc:1467] Processing CreateTablet for tablet bec27ccc8d2f4579aa8bab688ee018dd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d651ba6a15b04bb8a6b265db81f42706]), partition=
I20240501 20:10:01.878727 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec27ccc8d2f4579aa8bab688ee018dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.879693 23691 tablet_service.cc:1467] Processing CreateTablet for tablet bec27ccc8d2f4579aa8bab688ee018dd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d651ba6a15b04bb8a6b265db81f42706]), partition=
I20240501 20:10:01.880136 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec27ccc8d2f4579aa8bab688ee018dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.883953 23957 tablet_service.cc:1467] Processing CreateTablet for tablet bec27ccc8d2f4579aa8bab688ee018dd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d651ba6a15b04bb8a6b265db81f42706]), partition=
I20240501 20:10:01.884301 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec27ccc8d2f4579aa8bab688ee018dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.889941 27483 tablet_bootstrap.cc:492] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:01.892133 27483 tablet_bootstrap.cc:654] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.897722 27482 tablet_bootstrap.cc:492] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:01.899863 27482 tablet_bootstrap.cc:654] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.902105 27536 tablet_bootstrap.cc:492] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:01.902549 27482 tablet_bootstrap.cc:492] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:10:01.902714 27482 ts_tablet_manager.cc:1367] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:10:01.903532 27482 raft_consensus.cc:348] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:01.903820 27482 raft_consensus.cc:374] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.903928 27482 raft_consensus.cc:729] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.904346 27536 tablet_bootstrap.cc:654] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.904261 27482 consensus_queue.cc:260] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:01.905109 27482 ts_tablet_manager.cc:1398] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:01.916749 27536 tablet_bootstrap.cc:492] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:01.917003 27536 ts_tablet_manager.cc:1367] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240501 20:10:01.917814 27536 raft_consensus.cc:348] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:01.918126 27536 raft_consensus.cc:374] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.918260 27536 raft_consensus.cc:729] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.918543 27536 consensus_queue.cc:260] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:01.919186 27536 ts_tablet_manager.cc:1398] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:01.918561 27483 tablet_bootstrap.cc:492] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:01.931849 27483 ts_tablet_manager.cc:1367] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.012s
I20240501 20:10:01.932746 27483 raft_consensus.cc:348] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:01.933089 27483 raft_consensus.cc:374] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.933200 27483 raft_consensus.cc:729] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.933458 27483 consensus_queue.cc:260] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:01.960686 27483 ts_tablet_manager.cc:1398] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
I20240501 20:10:02.028999 27021 raft_consensus.cc:482] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:02.029210 27021 raft_consensus.cc:504] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.029933 27021 leader_election.cc:288] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:02.033130 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec27ccc8d2f4579aa8bab688ee018dd" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:10:02.033380 23846 raft_consensus.cc:2454] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:10:02.034936 23647 leader_election.cc:302] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:10:02.035496 27386 raft_consensus.cc:2781] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:02.035624 27386 raft_consensus.cc:482] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:02.035704 27386 raft_consensus.cc:3037] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.038369 27386 raft_consensus.cc:504] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.040827 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec27ccc8d2f4579aa8bab688ee018dd" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:10:02.041057 23976 raft_consensus.cc:3037] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.044584 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec27ccc8d2f4579aa8bab688ee018dd" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:10:02.050056 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec27ccc8d2f4579aa8bab688ee018dd" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:10:02.050285 23843 raft_consensus.cc:3037] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.053079 23843 raft_consensus.cc:2454] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:10:02.063753 27386 leader_election.cc:288] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:02.065166 23647 leader_election.cc:302] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:10:02.068921 27021 raft_consensus.cc:2781] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:02.069067 27021 raft_consensus.cc:686] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:10:02.069347 27021 consensus_queue.cc:237] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.085909 23976 raft_consensus.cc:2454] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:10:02.109591 26720 consensus_queue.cc:1035] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.116261 20820 catalog_manager.cc:5484] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:02.137092 26720 consensus_queue.cc:1035] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.218669 27510 consensus_queue.cc:1035] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.231905 21206 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:02.253476 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:02.269120 23689 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:02.270960 27510 consensus_queue.cc:1035] T 6d11505bdd2243949a9776989410e534 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.301826 27518 consensus_queue.cc:1035] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.328608 23846 raft_consensus.cc:1261] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:02.329181 27548 consensus_queue.cc:1035] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.331390 23976 raft_consensus.cc:1261] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:02.334789 27548 consensus_queue.cc:1035] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.338557 27518 consensus_queue.cc:1035] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.355029 20821 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet bec27ccc8d2f4579aa8bab688ee018dd (table rebalancer_dynamic_table [id=d651ba6a15b04bb8a6b265db81f42706]). Expected version 1 got 0
I20240501 20:10:02.362268 27570 mvcc.cc:205] Tried to move back new op lower bound from 7022977852731445248 to 7022977851678601216. Current Snapshot: MvccSnapshot[applied={T|T < 7022977852731445248}]
I20240501 20:10:02.366408 27585 tablet.cc:1722] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:02.368453 27585 tablet.cc:1620] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:10:02.369225 20821 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet bec27ccc8d2f4579aa8bab688ee018dd (table rebalancer_dynamic_table [id=d651ba6a15b04bb8a6b265db81f42706]). Expected version 1 got 0
I20240501 20:10:02.375308 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:02.387285 24089 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:02.390233 23689 tablet_service.cc:1467] Processing CreateTablet for tablet c58dd83ebd0949dea6f4567067c8dfc9 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.390738 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58dd83ebd0949dea6f4567067c8dfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.392346 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 62e27bcc1cf34d8ab5d50331bee12886 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.392810 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e27bcc1cf34d8ab5d50331bee12886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.396005 21206 tablet_service.cc:1467] Processing CreateTablet for tablet c58dd83ebd0949dea6f4567067c8dfc9 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.396504 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58dd83ebd0949dea6f4567067c8dfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.398135 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 62e27bcc1cf34d8ab5d50331bee12886 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.398505 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e27bcc1cf34d8ab5d50331bee12886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.398572 24089 tablet_service.cc:1467] Processing CreateTablet for tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.398975 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.398135 21208 tablet_service.cc:1467] Processing CreateTablet for tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.399883 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.434314 23824 tablet_service.cc:1467] Processing CreateTablet for tablet c58dd83ebd0949dea6f4567067c8dfc9 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.434929 23825 tablet_service.cc:1467] Processing CreateTablet for tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.437701 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58dd83ebd0949dea6f4567067c8dfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.438150 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.447160 27592 tablet_bootstrap.cc:492] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:02.451879 21337 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 20:10:02.453148 27592 tablet_bootstrap.cc:654] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.466728 20820 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=d651ba6a15b04bb8a6b265db81f42706] alter complete (version 1)
I20240501 20:10:02.470660 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:02.473891 27592 tablet_bootstrap.cc:492] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:10:02.474170 27592 ts_tablet_manager.cc:1367] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240501 20:10:02.487845 27592 raft_consensus.cc:348] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.490111 27592 raft_consensus.cc:374] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.490346 27592 raft_consensus.cc:729] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.490742 27592 consensus_queue.cc:260] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.492756 27581 tablet.cc:1722] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:02.493077 27592 ts_tablet_manager.cc:1398] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240501 20:10:02.493407 27581 tablet.cc:1620] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:10:02.495925 27591 tablet_bootstrap.cc:492] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:02.497560 27591 tablet_bootstrap.cc:654] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.499796 27592 tablet_bootstrap.cc:492] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:02.501861 27592 tablet_bootstrap.cc:654] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.512646 27596 tablet_bootstrap.cc:492] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:02.515823 27596 tablet_bootstrap.cc:654] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.517982 27594 tablet_bootstrap.cc:492] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:02.519905 27594 tablet_bootstrap.cc:654] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.546514 23825 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:02.557405 20947 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:10:02.567781 20820 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet bec27ccc8d2f4579aa8bab688ee018dd (table rebalancer_dynamic_table [id=d651ba6a15b04bb8a6b265db81f42706]). Expected version 1 got 0
I20240501 20:10:02.580403 27596 tablet_bootstrap.cc:492] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:10:02.584059 27594 tablet_bootstrap.cc:492] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:10:02.584313 27594 ts_tablet_manager.cc:1367] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20240501 20:10:02.585201 27594 raft_consensus.cc:348] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.585491 27594 raft_consensus.cc:374] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.586490 20821 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:02.587339 27596 ts_tablet_manager.cc:1367] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.000s
I20240501 20:10:02.588541 27596 raft_consensus.cc:348] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.589979 27594 raft_consensus.cc:729] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.590286 27594 consensus_queue.cc:260] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.591790 27596 raft_consensus.cc:374] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.591951 27596 raft_consensus.cc:729] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.592259 27596 consensus_queue.cc:260] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.592922 27596 ts_tablet_manager.cc:1398] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:10:02.598641 27594 ts_tablet_manager.cc:1398] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:10:02.598661 27592 tablet_bootstrap.cc:492] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:10:02.598829 27592 ts_tablet_manager.cc:1367] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.099s user 0.012s sys 0.010s
I20240501 20:10:02.599696 27592 raft_consensus.cc:348] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.600001 27592 raft_consensus.cc:374] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.600117 27592 raft_consensus.cc:729] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.600400 27592 consensus_queue.cc:260] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.602258 27592 ts_tablet_manager.cc:1398] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:02.632021 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 62e27bcc1cf34d8ab5d50331bee12886 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.632511 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e27bcc1cf34d8ab5d50331bee12886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.655618 27595 tablet_bootstrap.cc:492] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:02.660370 27581 tablet.cc:1722] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:02.660964 27581 tablet.cc:1620] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:10:02.661118 27585 tablet.cc:1722] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:02.661614 27585 tablet.cc:1620] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:10:02.668926 27591 tablet_bootstrap.cc:492] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:10:02.669086 27591 ts_tablet_manager.cc:1367] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.173s user 0.011s sys 0.004s
I20240501 20:10:02.669884 27591 raft_consensus.cc:348] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.670083 27595 tablet_bootstrap.cc:654] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.670238 27591 raft_consensus.cc:374] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.670360 27591 raft_consensus.cc:729] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.670656 27591 consensus_queue.cc:260] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.671245 27591 ts_tablet_manager.cc:1398] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:02.671380 27505 raft_consensus.cc:482] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:02.671618 27591 tablet_bootstrap.cc:492] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:02.671581 27505 raft_consensus.cc:504] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.673460 27591 tablet_bootstrap.cc:654] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.675767 20826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=d651ba6a15b04bb8a6b265db81f42706] alter complete (version 2)
I20240501 20:10:02.682680 27605 tablet_bootstrap.cc:492] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:02.689971 27605 tablet_bootstrap.cc:654] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.690037 27587 raft_consensus.cc:482] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:02.690219 27587 raft_consensus.cc:504] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.691762 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e27bcc1cf34d8ab5d50331bee12886" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:10:02.691994 21357 raft_consensus.cc:2454] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:10:02.692528 21162 leader_election.cc:302] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:10:02.692858 26629 raft_consensus.cc:2781] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:02.692939 26629 raft_consensus.cc:482] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:02.692992 26629 raft_consensus.cc:3037] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.696219 27604 tablet.cc:1722] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:02.697182 27595 tablet_bootstrap.cc:492] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:02.697348 27595 ts_tablet_manager.cc:1367] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.180s user 0.004s sys 0.000s
I20240501 20:10:02.698179 27595 raft_consensus.cc:348] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.698390 27595 raft_consensus.cc:374] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.695531 26629 raft_consensus.cc:504] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.698468 27595 raft_consensus.cc:729] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.698678 27595 consensus_queue.cc:260] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.699273 27595 ts_tablet_manager.cc:1398] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:02.699815 26629 leader_election.cc:288] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:02.700124 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e27bcc1cf34d8ab5d50331bee12886" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:10:02.700304 21357 raft_consensus.cc:3037] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.703377 21357 raft_consensus.cc:2454] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:10:02.704243 21162 leader_election.cc:302] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:10:02.704597 26629 raft_consensus.cc:2781] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:02.704823 26629 raft_consensus.cc:686] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:10:02.706347 26629 consensus_queue.cc:237] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.708097 27587 leader_election.cc:288] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:02.708369 27505 leader_election.cc:288] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:02.709499 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
W20240501 20:10:02.710145 23782 leader_election.cc:341] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:02.711017 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:02.711289 23709 raft_consensus.cc:2454] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:10:02.711665 27591 tablet_bootstrap.cc:492] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:10:02.711711 23781 leader_election.cc:302] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:10:02.711807 27591 ts_tablet_manager.cc:1367] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240501 20:10:02.712117 27505 raft_consensus.cc:2781] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:02.712235 27505 raft_consensus.cc:482] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:02.712319 27505 raft_consensus.cc:3037] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.713541 27591 raft_consensus.cc:348] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.714221 27591 raft_consensus.cc:374] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.715008 27505 raft_consensus.cc:504] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.714286 27591 raft_consensus.cc:729] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.718308 27591 consensus_queue.cc:260] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.718991 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:10:02.719492 27591 ts_tablet_manager.cc:1398] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20240501 20:10:02.719558 23782 leader_election.cc:341] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:02.720403 27505 leader_election.cc:288] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:02.720762 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:10:02.720949 23709 raft_consensus.cc:3037] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.721161 27591 tablet_bootstrap.cc:492] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:02.726858 23709 raft_consensus.cc:2454] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:10:02.727393 23781 leader_election.cc:302] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:10:02.728099 27505 raft_consensus.cc:2781] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:02.728235 27505 raft_consensus.cc:686] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:10:02.728602 27505 consensus_queue.cc:237] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.739005 27591 tablet_bootstrap.cc:654] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.737054 20821 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet bec27ccc8d2f4579aa8bab688ee018dd (table rebalancer_dynamic_table [id=d651ba6a15b04bb8a6b265db81f42706]). Expected version 2 got 0
I20240501 20:10:02.710685 23974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e27bcc1cf34d8ab5d50331bee12886" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
W20240501 20:10:02.747257 21161 leader_election.cc:341] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:02.716306 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e27bcc1cf34d8ab5d50331bee12886" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:10:02.742980 23957 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240501 20:10:02.748632 21161 leader_election.cc:341] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:02.750283 27591 tablet_bootstrap.cc:492] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:10:02.751763 27591 ts_tablet_manager.cc:1367] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.003s
I20240501 20:10:02.752583 27591 raft_consensus.cc:348] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.758713 27591 raft_consensus.cc:374] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.759039 27591 raft_consensus.cc:729] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.698005 27604 tablet.cc:1620] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:10:02.773116 20821 catalog_manager.cc:5484] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:02.773696 27591 consensus_queue.cc:260] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.776213 27591 ts_tablet_manager.cc:1398] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240501 20:10:02.779405 26720 raft_consensus.cc:482] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:02.779660 26720 raft_consensus.cc:504] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.780560 26720 leader_election.cc:288] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:02.780875 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad72fa9eb71c4a2f9eaee8f48f7bbca9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:10:02.781083 21228 raft_consensus.cc:2454] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:10:02.781520 23782 leader_election.cc:302] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:10:02.782459 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad72fa9eb71c4a2f9eaee8f48f7bbca9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:10:02.782820 27505 raft_consensus.cc:2781] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:02.782966 27505 raft_consensus.cc:482] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:02.783063 27505 raft_consensus.cc:3037] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.788249 24110 raft_consensus.cc:2454] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 0.
I20240501 20:10:02.790994 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:02.794759 27505 raft_consensus.cc:504] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.796240 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad72fa9eb71c4a2f9eaee8f48f7bbca9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:10:02.796433 21228 raft_consensus.cc:3037] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.799217 21228 raft_consensus.cc:2454] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:10:02.802543 23782 leader_election.cc:302] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:10:02.804242 26720 raft_consensus.cc:2781] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:02.843945 27581 tablet.cc:1722] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:02.861128 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=d651ba6a15b04bb8a6b265db81f42706] alter complete (version 3)
I20240501 20:10:02.872349 20821 catalog_manager.cc:5484] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:02.877290 27605 tablet_bootstrap.cc:492] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:02.877597 27605 ts_tablet_manager.cc:1367] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.198s user 0.028s sys 0.009s
I20240501 20:10:02.878568 27605 raft_consensus.cc:348] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.878918 27605 raft_consensus.cc:374] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.878980 27505 leader_election.cc:288] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:02.879194 27605 raft_consensus.cc:729] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.879472 27605 consensus_queue.cc:260] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:02.880321 27605 ts_tablet_manager.cc:1398] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:02.899720 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad72fa9eb71c4a2f9eaee8f48f7bbca9" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:10:02.899983 24110 raft_consensus.cc:3037] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.901280 20820 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet bec27ccc8d2f4579aa8bab688ee018dd (table rebalancer_dynamic_table_new_name [id=d651ba6a15b04bb8a6b265db81f42706]). Expected version 3 got 1
I20240501 20:10:02.905644 27587 raft_consensus.cc:482] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:02.905786 27587 raft_consensus.cc:504] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
W20240501 20:10:02.906262 23779 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 94653.4us
I20240501 20:10:02.906543 27587 leader_election.cc:288] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:02.907686 24110 raft_consensus.cc:2454] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:10:02.907925 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:10:02.908332 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:02.908587 23709 raft_consensus.cc:2379] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4af096f5a6a9440a94f6682b399efe20 in current term 1: Already voted for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in this term.
I20240501 20:10:02.910305 21163 leader_election.cc:302] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:10:02.910691 27587 raft_consensus.cc:3037] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.911645 26720 raft_consensus.cc:686] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:10:02.911739 27604 tablet.cc:1722] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:02.912191 26720 consensus_queue.cc:237] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:02.912331 27604 tablet.cc:1620] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:10:02.913499 27587 raft_consensus.cc:2726] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:02.920430 27585 tablet.cc:1722] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:02.922783 20821 catalog_manager.cc:5484] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:02.927011 20826 catalog_manager.cc:5576] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019) does not have the latest schema for tablet bec27ccc8d2f4579aa8bab688ee018dd (table rebalancer_dynamic_table_new_name [id=d651ba6a15b04bb8a6b265db81f42706]). Expected version 3 got 2
I20240501 20:10:02.930151 27604 tablet.cc:1722] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:02.973083 20821 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:02.973323 20821 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:02.976425 20821 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet bec27ccc8d2f4579aa8bab688ee018dd
I20240501 20:10:02.977485 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet bec27ccc8d2f4579aa8bab688ee018dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:10:02.977653 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet bec27ccc8d2f4579aa8bab688ee018dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:02.978209 27619 tablet_replica.cc:330] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:10:02.978379 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet bec27ccc8d2f4579aa8bab688ee018dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:02.978514 27619 raft_consensus.cc:2229] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.978729 27619 raft_consensus.cc:2258] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.981547 27619 ts_tablet_manager.cc:1875] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.991710 27619 ts_tablet_manager.cc:1888] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:02.991961 27619 log.cc:1196] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/bec27ccc8d2f4579aa8bab688ee018dd
I20240501 20:10:02.992496 27619 ts_tablet_manager.cc:1909] T bec27ccc8d2f4579aa8bab688ee018dd P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:10:02.993655 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet bec27ccc8d2f4579aa8bab688ee018dd (table rebalancer_dynamic_table_new_name [id=d651ba6a15b04bb8a6b265db81f42706]) successfully deleted
I20240501 20:10:02.995934 27621 tablet_replica.cc:330] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:02.996227 27621 raft_consensus.cc:2229] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.996663 27621 raft_consensus.cc:2258] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.997100 27620 tablet_replica.cc:330] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:02.997301 27620 raft_consensus.cc:2229] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:02.997630 27620 raft_consensus.cc:2258] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.998524 27620 ts_tablet_manager.cc:1875] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.000341 27621 ts_tablet_manager.cc:1875] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.006742 27621 ts_tablet_manager.cc:1888] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:03.007175 27621 log.cc:1196] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/bec27ccc8d2f4579aa8bab688ee018dd
I20240501 20:10:03.007807 27621 ts_tablet_manager.cc:1909] T bec27ccc8d2f4579aa8bab688ee018dd P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:10:03.008533 27620 ts_tablet_manager.cc:1888] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:03.008667 27620 log.cc:1196] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/bec27ccc8d2f4579aa8bab688ee018dd
I20240501 20:10:03.009052 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet bec27ccc8d2f4579aa8bab688ee018dd (table rebalancer_dynamic_table_new_name [id=d651ba6a15b04bb8a6b265db81f42706]) successfully deleted
I20240501 20:10:03.009502 27620 ts_tablet_manager.cc:1909] T bec27ccc8d2f4579aa8bab688ee018dd P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:10:03.010841 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet bec27ccc8d2f4579aa8bab688ee018dd (table rebalancer_dynamic_table_new_name [id=d651ba6a15b04bb8a6b265db81f42706]) successfully deleted
I20240501 20:10:03.080438 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:03.089694 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 63d0d157748642118fd7169868dc8cb7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d998094eaec2402690df55f2026b9bb9]), partition=
I20240501 20:10:03.089792 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 63d0d157748642118fd7169868dc8cb7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d998094eaec2402690df55f2026b9bb9]), partition=
I20240501 20:10:03.090101 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d0d157748642118fd7169868dc8cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.090137 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d0d157748642118fd7169868dc8cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.094724 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 63d0d157748642118fd7169868dc8cb7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d998094eaec2402690df55f2026b9bb9]), partition=
I20240501 20:10:03.094756 27634 tablet_bootstrap.cc:492] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:03.095096 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d0d157748642118fd7169868dc8cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.096521 27634 tablet_bootstrap.cc:654] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.103813 27595 tablet_bootstrap.cc:492] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:03.103533 27605 tablet_bootstrap.cc:492] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:03.105440 27595 tablet_bootstrap.cc:654] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.105439 27605 tablet_bootstrap.cc:654] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.109161 27605 tablet_bootstrap.cc:492] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:03.109424 27605 ts_tablet_manager.cc:1367] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:10:03.110395 27605 raft_consensus.cc:348] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:03.110718 27605 raft_consensus.cc:374] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.110852 27605 raft_consensus.cc:729] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.112524 27505 consensus_queue.cc:1035] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.111716 27605 consensus_queue.cc:260] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:03.114084 27605 ts_tablet_manager.cc:1398] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:03.128217 27595 tablet_bootstrap.cc:492] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:03.128417 27595 ts_tablet_manager.cc:1367] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240501 20:10:03.129232 27595 raft_consensus.cc:348] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:03.129468 27595 raft_consensus.cc:374] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.129527 27595 raft_consensus.cc:729] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.129747 27595 consensus_queue.cc:260] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:03.130649 27595 ts_tablet_manager.cc:1398] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:03.136349 23842 consensus_queue.cc:237] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:03.141585 21228 raft_consensus.cc:1261] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:03.141966 21098 raft_consensus.cc:1261] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:03.142601 26720 consensus_queue.cc:1035] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:03.144646 27505 consensus_queue.cc:1035] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:03.151872 27613 raft_consensus.cc:2932] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:03.153067 21228 raft_consensus.cc:2932] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:03.158679 26720 consensus_queue.cc:1035] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.161202 20821 catalog_manager.cc:5484] T 7a23f60c87794a49ae1f82dff06fe881 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:03.161913 27461 raft_consensus.cc:482] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.162163 27461 raft_consensus.cc:504] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:03.162982 27461 leader_election.cc:288] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:03.164930 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d0d157748642118fd7169868dc8cb7" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
W20240501 20:10:03.167465 23909 leader_election.cc:341] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:03.168121 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a23f60c87794a49ae1f82dff06fe881 with delete_type TABLET_DATA_TOMBSTONED (TS 8559229e044b4e1ea661b881b3f9af50 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:03.170123 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d0d157748642118fd7169868dc8cb7" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:03.170401 23712 raft_consensus.cc:2454] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:10:03.173960 23911 leader_election.cc:302] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters: cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:03.176374 27461 raft_consensus.cc:2781] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:03.176555 27461 raft_consensus.cc:482] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:03.176659 27461 raft_consensus.cc:3037] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.182441 27461 raft_consensus.cc:504] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:03.184383 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d0d157748642118fd7169868dc8cb7" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:10:03.184659 23710 raft_consensus.cc:3037] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.187369 23710 raft_consensus.cc:2454] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:10:03.187567 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d0d157748642118fd7169868dc8cb7" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
W20240501 20:10:03.188323 23909 leader_election.cc:341] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:03.188630 23911 leader_election.cc:302] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters: cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:03.188979 27462 raft_consensus.cc:2781] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:03.189018 27648 tablet_replica.cc:330] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:03.189307 27648 raft_consensus.cc:2229] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.189582 27648 raft_consensus.cc:2258] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.190223 27462 raft_consensus.cc:686] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:10:03.190433 27648 ts_tablet_manager.cc:1875] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:03.190537 27462 consensus_queue.cc:237] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:03.193567 27461 leader_election.cc:288] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:03.203199 20821 catalog_manager.cc:5484] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:03.211118 27634 tablet_bootstrap.cc:492] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:10:03.211812 27634 ts_tablet_manager.cc:1367] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.117s user 0.029s sys 0.034s
I20240501 20:10:03.212421 21098 raft_consensus.cc:2932] T 7a23f60c87794a49ae1f82dff06fe881 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:03.215052 27634 raft_consensus.cc:348] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:03.215562 27634 raft_consensus.cc:374] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.215799 27634 raft_consensus.cc:729] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.216722 27634 consensus_queue.cc:260] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:03.217763 27634 ts_tablet_manager.cc:1398] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:03.223824 27648 ts_tablet_manager.cc:1888] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:03.224031 27648 log.cc:1196] T 7a23f60c87794a49ae1f82dff06fe881 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/7a23f60c87794a49ae1f82dff06fe881
I20240501 20:10:03.225322 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 7a23f60c87794a49ae1f82dff06fe881 (table rebalancer_extra_table_0 [id=bd28afd6621f41058003d0850fb729db]) successfully deleted
I20240501 20:10:03.228072 27587 consensus_queue.cc:1035] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.235867 23977 raft_consensus.cc:3037] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.238890 26629 consensus_queue.cc:1035] T 62e27bcc1cf34d8ab5d50331bee12886 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.315369 20820 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:03.322273 20964 raft_consensus.cc:3037] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.322461 23709 raft_consensus.cc:1261] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:03.323107 27462 consensus_queue.cc:1035] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.326300 20964 raft_consensus.cc:1261] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:03.329674 27461 consensus_queue.cc:1035] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.333726 27577 mvcc.cc:205] Tried to move back new op lower bound from 7022977856802713600 to 7022977856271941632. Current Snapshot: MvccSnapshot[applied={T|T < 7022977856802713600}]
I20240501 20:10:03.334199 27604 tablet.cc:1722] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:03.334715 27604 tablet.cc:1620] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:10:03.334614 27581 tablet.cc:1722] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:03.334841 20821 catalog_manager.cc:5576] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155) does not have the latest schema for tablet 63d0d157748642118fd7169868dc8cb7 (table rebalancer_dynamic_table [id=d998094eaec2402690df55f2026b9bb9]). Expected version 1 got 0
I20240501 20:10:03.335083 27581 tablet.cc:1620] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:10:03.342689 27613 consensus_queue.cc:1035] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.356374 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=d998094eaec2402690df55f2026b9bb9] alter complete (version 1)
I20240501 20:10:03.366097 26720 consensus_queue.cc:1035] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.381280 20821 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 63d0d157748642118fd7169868dc8cb7 (table rebalancer_dynamic_table [id=d998094eaec2402690df55f2026b9bb9]). Expected version 1 got 0
I20240501 20:10:03.388178 27683 tablet.cc:1722] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:03.389539 27683 tablet.cc:1620] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:10:03.393940 20821 catalog_manager.cc:5576] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259) does not have the latest schema for tablet 63d0d157748642118fd7169868dc8cb7 (table rebalancer_dynamic_table [id=d998094eaec2402690df55f2026b9bb9]). Expected version 1 got 0
I20240501 20:10:03.478144 20824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:03.504393 27604 tablet.cc:1722] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:03.504979 27604 tablet.cc:1620] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: MemRowSet was empty: no flush needed.
I20240501 20:10:03.506695 27683 tablet.cc:1722] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:03.507181 27683 tablet.cc:1620] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: MemRowSet was empty: no flush needed.
I20240501 20:10:03.512459 27581 tablet.cc:1722] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:03.512949 27581 tablet.cc:1620] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: MemRowSet was empty: no flush needed.
I20240501 20:10:03.534597 20818 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:03.556313 20824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=d998094eaec2402690df55f2026b9bb9] alter complete (version 2)
I20240501 20:10:03.570441 23691 tablet_service.cc:1467] Processing CreateTablet for tablet d7222b721f344c6fab6a494480c9d560 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=8b61e91f38c241d8b8dbf05d1ecbcd22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.570940 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7222b721f344c6fab6a494480c9d560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.571748 23825 tablet_service.cc:1467] Processing CreateTablet for tablet d7222b721f344c6fab6a494480c9d560 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=8b61e91f38c241d8b8dbf05d1ecbcd22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.571966 23824 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed52ae966604421beb3ca1f381d99b3 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=8b61e91f38c241d8b8dbf05d1ecbcd22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.572191 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7222b721f344c6fab6a494480c9d560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.572446 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed52ae966604421beb3ca1f381d99b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.576972 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed52ae966604421beb3ca1f381d99b3 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=8b61e91f38c241d8b8dbf05d1ecbcd22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.577477 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed52ae966604421beb3ca1f381d99b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.577883 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 57cda4be1d824e629b2938d44c5f2398 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=8b61e91f38c241d8b8dbf05d1ecbcd22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.578307 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cda4be1d824e629b2938d44c5f2398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.578581 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 57cda4be1d824e629b2938d44c5f2398 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=8b61e91f38c241d8b8dbf05d1ecbcd22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.578954 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cda4be1d824e629b2938d44c5f2398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.579573 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 57cda4be1d824e629b2938d44c5f2398 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=8b61e91f38c241d8b8dbf05d1ecbcd22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.580044 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cda4be1d824e629b2938d44c5f2398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.576962 24089 tablet_service.cc:1467] Processing CreateTablet for tablet d7222b721f344c6fab6a494480c9d560 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=8b61e91f38c241d8b8dbf05d1ecbcd22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.580351 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7222b721f344c6fab6a494480c9d560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.583951 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed52ae966604421beb3ca1f381d99b3 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=8b61e91f38c241d8b8dbf05d1ecbcd22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.584374 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed52ae966604421beb3ca1f381d99b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.593170 27605 tablet_bootstrap.cc:492] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:03.594981 27605 tablet_bootstrap.cc:654] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.604525 27595 tablet_bootstrap.cc:492] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:03.606684 27595 tablet_bootstrap.cc:654] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.616557 27699 tablet_bootstrap.cc:492] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:03.618225 27605 tablet_bootstrap.cc:492] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:03.618510 27605 ts_tablet_manager.cc:1367] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20240501 20:10:03.619448 27605 raft_consensus.cc:348] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:03.619781 27605 raft_consensus.cc:374] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.619823 27595 tablet_bootstrap.cc:492] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:03.619947 27605 raft_consensus.cc:729] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.619980 27595 ts_tablet_manager.cc:1367] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20240501 20:10:03.620299 27605 consensus_queue.cc:260] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:03.620834 27595 raft_consensus.cc:348] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:03.621111 27595 raft_consensus.cc:374] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.621268 27595 raft_consensus.cc:729] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.621568 27595 consensus_queue.cc:260] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:03.622483 27595 ts_tablet_manager.cc:1398] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:03.625274 27605 ts_tablet_manager.cc:1398] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:10:03.628979 27605 tablet_bootstrap.cc:492] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:03.630126 27698 tablet_bootstrap.cc:492] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:03.631711 27696 tablet_bootstrap.cc:492] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:03.633216 27605 tablet_bootstrap.cc:654] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.634110 27698 tablet_bootstrap.cc:654] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.636281 27699 tablet_bootstrap.cc:654] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.645977 27605 tablet_bootstrap.cc:492] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:03.646646 27696 tablet_bootstrap.cc:654] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.646685 27605 ts_tablet_manager.cc:1367] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20240501 20:10:03.648089 27605 raft_consensus.cc:348] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:03.648581 27605 raft_consensus.cc:374] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.648703 27605 raft_consensus.cc:729] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.649058 27605 consensus_queue.cc:260] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:03.651314 27605 ts_tablet_manager.cc:1398] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:10:03.652395 27699 tablet_bootstrap.cc:492] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:10:03.652664 27699 ts_tablet_manager.cc:1367] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.000s
I20240501 20:10:03.653844 27699 raft_consensus.cc:348] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:03.654207 27699 raft_consensus.cc:374] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.654348 27699 raft_consensus.cc:729] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.654639 27699 consensus_queue.cc:260] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:03.656647 27699 ts_tablet_manager.cc:1398] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:03.656994 27699 tablet_bootstrap.cc:492] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:03.662102 20819 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:03.663529 27699 tablet_bootstrap.cc:654] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.665658 27547 raft_consensus.cc:482] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.665830 27547 raft_consensus.cc:504] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:03.666674 27547 leader_election.cc:288] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:03.667973 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7222b721f344c6fab6a494480c9d560" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:10:03.667956 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7222b721f344c6fab6a494480c9d560" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:10:03.668185 24110 raft_consensus.cc:2454] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:10:03.668784 23646 leader_election.cc:302] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
W20240501 20:10:03.669453 23647 leader_election.cc:341] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:03.670315 27547 raft_consensus.cc:2781] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:03.670439 27547 raft_consensus.cc:482] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:03.670516 27547 raft_consensus.cc:3037] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.672355 27708 tablet_bootstrap.cc:492] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:03.673520 27547 raft_consensus.cc:504] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:03.674641 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7222b721f344c6fab6a494480c9d560" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:10:03.674860 24110 raft_consensus.cc:3037] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.676082 27708 tablet_bootstrap.cc:654] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.677647 24110 raft_consensus.cc:2454] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:10:03.678247 23646 leader_election.cc:302] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:10:03.678587 27600 raft_consensus.cc:2781] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:03.681284 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7222b721f344c6fab6a494480c9d560" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:10:03.681342 27547 leader_election.cc:288] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
W20240501 20:10:03.681922 23647 leader_election.cc:341] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:03.682180 27604 tablet.cc:1722] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:03.685173 27683 tablet.cc:1722] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:03.686194 27581 tablet.cc:1722] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:03.697263 27600 raft_consensus.cc:686] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:10:03.698633 27600 consensus_queue.cc:237] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:03.701367 20819 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=d998094eaec2402690df55f2026b9bb9] alter complete (version 3)
I20240501 20:10:03.704841 27699 tablet_bootstrap.cc:492] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:10:03.705116 27699 ts_tablet_manager.cc:1367] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.000s
I20240501 20:10:03.706037 27699 raft_consensus.cc:348] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:03.706737 27699 raft_consensus.cc:374] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.707008 27699 raft_consensus.cc:729] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.707245 27696 tablet_bootstrap.cc:492] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:10:03.707481 27696 ts_tablet_manager.cc:1367] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.000s
I20240501 20:10:03.707531 27699 consensus_queue.cc:260] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:03.708403 27696 raft_consensus.cc:348] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:03.708693 27696 raft_consensus.cc:374] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.708832 27696 raft_consensus.cc:729] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.709132 27696 consensus_queue.cc:260] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:03.710963 27699 ts_tablet_manager.cc:1398] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:03.711148 27696 ts_tablet_manager.cc:1398] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:03.743750 27708 tablet_bootstrap.cc:492] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:10:03.743992 27708 ts_tablet_manager.cc:1367] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.016s
I20240501 20:10:03.744728 27708 raft_consensus.cc:348] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:03.745041 27708 raft_consensus.cc:374] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.745154 27708 raft_consensus.cc:729] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.745435 27708 consensus_queue.cc:260] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:03.747599 27708 ts_tablet_manager.cc:1398] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:03.752681 27698 tablet_bootstrap.cc:492] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:10:03.753131 27698 ts_tablet_manager.cc:1367] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.126s user 0.005s sys 0.000s
I20240501 20:10:03.756289 27698 raft_consensus.cc:348] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:03.756618 27698 raft_consensus.cc:374] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.757526 27698 raft_consensus.cc:729] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.757889 27698 consensus_queue.cc:260] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:03.761164 27698 ts_tablet_manager.cc:1398] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 20:10:03.764055 27698 tablet_bootstrap.cc:492] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:03.764175 27345 raft_consensus.cc:482] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.764376 27345 raft_consensus.cc:504] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:03.765444 27345 leader_election.cc:288] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:03.766389 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cda4be1d824e629b2938d44c5f2398" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:10:03.766670 21227 raft_consensus.cc:2454] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:10:03.767652 21294 leader_election.cc:302] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:10:03.767709 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cda4be1d824e629b2938d44c5f2398" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:10:03.767935 23977 raft_consensus.cc:2454] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:10:03.768002 27345 raft_consensus.cc:2781] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:03.768070 27345 raft_consensus.cc:482] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:03.768152 27345 raft_consensus.cc:3037] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.770833 27345 raft_consensus.cc:504] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:03.771512 27345 leader_election.cc:288] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:03.772116 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cda4be1d824e629b2938d44c5f2398" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:10:03.772261 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cda4be1d824e629b2938d44c5f2398" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:10:03.772372 21227 raft_consensus.cc:3037] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.772487 23977 raft_consensus.cc:3037] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.772389 20820 catalog_manager.cc:5484] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:03.775144 21227 raft_consensus.cc:2454] T 57cda4be1d824e629b2938d44c5f2398 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:10:03.775161 23977 raft_consensus.cc:2454] T 57cda4be1d824e629b2938d44c5f2398 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:10:03.776856 21294 leader_election.cc:302] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:10:03.777279 27345 raft_consensus.cc:2781] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:03.777463 27345 raft_consensus.cc:686] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:10:03.777776 27345 consensus_queue.cc:237] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:03.784224 20824 catalog_manager.cc:5484] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:03.789343 27666 raft_consensus.cc:482] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.789500 27666 raft_consensus.cc:504] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:03.790098 27666 leader_election.cc:288] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:03.790501 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed52ae966604421beb3ca1f381d99b3" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:10:03.790743 23977 raft_consensus.cc:2454] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:10:03.791077 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed52ae966604421beb3ca1f381d99b3" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:10:03.785609 27698 tablet_bootstrap.cc:654] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.791508 24043 leader_election.cc:302] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters:
W20240501 20:10:03.791708 24046 leader_election.cc:341] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:03.791819 27666 raft_consensus.cc:2781] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:03.791940 27666 raft_consensus.cc:482] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:03.792042 27666 raft_consensus.cc:3037] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.795521 27666 raft_consensus.cc:504] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:03.796202 27666 leader_election.cc:288] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:03.796551 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed52ae966604421beb3ca1f381d99b3" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:10:03.796766 23977 raft_consensus.cc:3037] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.796777 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed52ae966604421beb3ca1f381d99b3" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
W20240501 20:10:03.797283 24046 leader_election.cc:341] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:03.798764 23977 raft_consensus.cc:2454] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:10:03.799139 24043 leader_election.cc:302] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50; no voters: c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:10:03.799408 27666 raft_consensus.cc:2781] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:03.799577 27666 raft_consensus.cc:686] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:10:03.800038 27666 consensus_queue.cc:237] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:03.803982 20824 catalog_manager.cc:5484] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:03.805233 20818 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:03.805436 20818 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:03.806041 27698 tablet_bootstrap.cc:492] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:10:03.809702 27698 ts_tablet_manager.cc:1367] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.003s
I20240501 20:10:03.810053 20818 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 63d0d157748642118fd7169868dc8cb7
I20240501 20:10:03.811816 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d0d157748642118fd7169868dc8cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:03.811836 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d0d157748642118fd7169868dc8cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:03.812708 27698 raft_consensus.cc:348] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:03.813053 27698 raft_consensus.cc:374] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.813184 27698 raft_consensus.cc:729] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.813484 27698 consensus_queue.cc:260] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:03.814842 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d0d157748642118fd7169868dc8cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:54238
I20240501 20:10:03.815061 27698 ts_tablet_manager.cc:1398] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:10:03.836378 27720 tablet_replica.cc:330] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:03.836911 27721 tablet_replica.cc:330] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:10:03.837327 27721 raft_consensus.cc:2229] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.837970 27719 tablet_replica.cc:330] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:03.838373 27720 raft_consensus.cc:2229] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:03.838711 27721 raft_consensus.cc:2258] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.839269 27720 raft_consensus.cc:2258] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.839788 27719 raft_consensus.cc:2229] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.841007 27721 ts_tablet_manager.cc:1875] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.843086 27719 raft_consensus.cc:2258] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.843756 27720 ts_tablet_manager.cc:1875] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.844703 27719 ts_tablet_manager.cc:1875] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.862851 27721 ts_tablet_manager.cc:1888] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:03.863204 27721 log.cc:1196] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/63d0d157748642118fd7169868dc8cb7
I20240501 20:10:03.863207 27719 ts_tablet_manager.cc:1888] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:03.863327 27719 log.cc:1196] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/63d0d157748642118fd7169868dc8cb7
I20240501 20:10:03.863806 27721 ts_tablet_manager.cc:1909] T 63d0d157748642118fd7169868dc8cb7 P cda19d6a81c0473db302535ce2ada7f0: Deleting consensus metadata
I20240501 20:10:03.863840 27719 ts_tablet_manager.cc:1909] T 63d0d157748642118fd7169868dc8cb7 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting consensus metadata
I20240501 20:10:03.866559 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 63d0d157748642118fd7169868dc8cb7 (table rebalancer_dynamic_table_new_name [id=d998094eaec2402690df55f2026b9bb9]) successfully deleted
I20240501 20:10:03.866597 27720 ts_tablet_manager.cc:1888] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:03.866712 27720 log.cc:1196] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/63d0d157748642118fd7169868dc8cb7
I20240501 20:10:03.867276 27720 ts_tablet_manager.cc:1909] T 63d0d157748642118fd7169868dc8cb7 P 744c5a532e724391830c0785d32eb2fb: Deleting consensus metadata
I20240501 20:10:03.868249 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 63d0d157748642118fd7169868dc8cb7 (table rebalancer_dynamic_table_new_name [id=d998094eaec2402690df55f2026b9bb9]) successfully deleted
I20240501 20:10:03.868588 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 63d0d157748642118fd7169868dc8cb7 (table rebalancer_dynamic_table_new_name [id=d998094eaec2402690df55f2026b9bb9]) successfully deleted
I20240501 20:10:03.915339 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:03.932405 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 869370236097417a9942c450b2b02c40 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2106faa2b9e4fb7b181571246c3df5b]), partition=
I20240501 20:10:03.932933 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869370236097417a9942c450b2b02c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.937320 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 869370236097417a9942c450b2b02c40 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2106faa2b9e4fb7b181571246c3df5b]), partition=
I20240501 20:10:03.937505 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 869370236097417a9942c450b2b02c40 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2106faa2b9e4fb7b181571246c3df5b]), partition=
I20240501 20:10:03.937723 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869370236097417a9942c450b2b02c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.937837 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869370236097417a9942c450b2b02c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.947517 27696 tablet_bootstrap.cc:492] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:03.949182 27696 tablet_bootstrap.cc:654] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.969863 27698 tablet_bootstrap.cc:492] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:03.971305 27696 tablet_bootstrap.cc:492] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:10:03.971473 27696 ts_tablet_manager.cc:1367] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.010s
I20240501 20:10:03.971704 27698 tablet_bootstrap.cc:654] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.973021 27696 raft_consensus.cc:348] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:03.973250 27696 raft_consensus.cc:374] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.973330 27696 raft_consensus.cc:729] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.985641 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:03.976779 27696 consensus_queue.cc:260] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:03.987793 27696 ts_tablet_manager.cc:1398] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240501 20:10:03.998916 21336 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:10:04.050451 27698 tablet_bootstrap.cc:492] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: No bootstrap required, opened a new log
I20240501 20:10:04.050724 27698 ts_tablet_manager.cc:1367] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.081s user 0.001s sys 0.008s
I20240501 20:10:04.051726 27698 raft_consensus.cc:348] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:04.052037 27698 raft_consensus.cc:374] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.052176 27698 raft_consensus.cc:729] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.052507 27698 consensus_queue.cc:260] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:04.057063 23957 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:04.057063 27698 ts_tablet_manager.cc:1398] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:04.067634 27734 tablet_bootstrap.cc:492] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:04.069792 23691 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:04.074787 27734 tablet_bootstrap.cc:654] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.105892 24089 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:04.114107 27734 tablet_bootstrap.cc:492] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:10:04.114415 27734 ts_tablet_manager.cc:1367] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.000s
I20240501 20:10:04.115258 27734 raft_consensus.cc:348] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:04.115659 27734 raft_consensus.cc:374] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.115765 27734 raft_consensus.cc:729] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.116086 27734 consensus_queue.cc:260] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:04.123468 27734 ts_tablet_manager.cc:1398] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:10:04.131067 20947 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:10:04.155666 27566 raft_consensus.cc:482] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.155833 27566 raft_consensus.cc:504] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:04.156491 27566 leader_election.cc:288] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:04.157335 23842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869370236097417a9942c450b2b02c40" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:10:04.157328 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869370236097417a9942c450b2b02c40" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:10:04.157564 21227 raft_consensus.cc:2454] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:10:04.157593 23842 raft_consensus.cc:2454] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 0.
I20240501 20:10:04.157997 27722 consensus_queue.cc:1035] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.158129 21034 leader_election.cc:302] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:04.159325 27566 raft_consensus.cc:2781] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:04.159405 27566 raft_consensus.cc:482] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:04.159464 27566 raft_consensus.cc:3037] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.167649 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:04.170528 27566 raft_consensus.cc:504] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:04.171685 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869370236097417a9942c450b2b02c40" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:10:04.171901 23843 raft_consensus.cc:3037] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.172153 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869370236097417a9942c450b2b02c40" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:10:04.172363 21228 raft_consensus.cc:3037] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.174841 23843 raft_consensus.cc:2454] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:10:04.174955 21228 raft_consensus.cc:2454] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:10:04.175397 21034 leader_election.cc:302] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:04.177505 27563 raft_consensus.cc:2781] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:04.185439 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:04.188720 27566 leader_election.cc:288] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 1 election: Requested vote from peers c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:04.191226 27722 consensus_queue.cc:1035] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.219264 26720 raft_consensus.cc:482] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.219456 26720 raft_consensus.cc:504] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:04.220180 26720 leader_election.cc:288] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:04.221119 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7222b721f344c6fab6a494480c9d560" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:04.221206 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7222b721f344c6fab6a494480c9d560" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:10:04.221498 24110 raft_consensus.cc:2379] T d7222b721f344c6fab6a494480c9d560 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in this term.
I20240501 20:10:04.230839 27563 raft_consensus.cc:686] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:10:04.231160 27600 consensus_queue.cc:1035] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.231388 27563 consensus_queue.cc:237] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:04.237239 23781 leader_election.cc:302] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:10:04.237792 26720 raft_consensus.cc:3037] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.243997 26720 raft_consensus.cc:2726] T d7222b721f344c6fab6a494480c9d560 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:04.251508 20818 catalog_manager.cc:5484] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:04.266417 27600 consensus_queue.cc:1035] T d7222b721f344c6fab6a494480c9d560 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.289153 27615 raft_consensus.cc:482] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.289355 27615 raft_consensus.cc:504] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:04.290314 27615 leader_election.cc:288] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:04.292502 23974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed52ae966604421beb3ca1f381d99b3" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:10:04.292757 23974 raft_consensus.cc:2379] T 4ed52ae966604421beb3ca1f381d99b3 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in current term 1: Already voted for candidate 8559229e044b4e1ea661b881b3f9af50 in this term.
I20240501 20:10:04.294762 24110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed52ae966604421beb3ca1f381d99b3" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:10:04.299998 23781 leader_election.cc:302] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:10:04.301347 27615 raft_consensus.cc:3037] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.305017 27615 raft_consensus.cc:2726] T 4ed52ae966604421beb3ca1f381d99b3 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:04.355288 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:04.366844 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 047bd48fda76496f820be523ad42dc5b (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c738d6165fed401e9f648ef1f8c6b97c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.367362 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047bd48fda76496f820be523ad42dc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.367957 21077 tablet_service.cc:1467] Processing CreateTablet for tablet 047bd48fda76496f820be523ad42dc5b (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c738d6165fed401e9f648ef1f8c6b97c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.367964 20947 tablet_service.cc:1467] Processing CreateTablet for tablet aefb41ceeb524a28a6c17db6b11316e2 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c738d6165fed401e9f648ef1f8c6b97c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.368355 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefb41ceeb524a28a6c17db6b11316e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.368371 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047bd48fda76496f820be523ad42dc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.369258 20945 tablet_service.cc:1467] Processing CreateTablet for tablet a79db73c177047d8bab730a6c42637ba (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c738d6165fed401e9f648ef1f8c6b97c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.369668 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79db73c177047d8bab730a6c42637ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.369621 21337 tablet_service.cc:1467] Processing CreateTablet for tablet aefb41ceeb524a28a6c17db6b11316e2 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c738d6165fed401e9f648ef1f8c6b97c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.369985 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefb41ceeb524a28a6c17db6b11316e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.373566 21338 tablet_service.cc:1467] Processing CreateTablet for tablet a79db73c177047d8bab730a6c42637ba (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c738d6165fed401e9f648ef1f8c6b97c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.373956 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79db73c177047d8bab730a6c42637ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.374049 21207 tablet_service.cc:1467] Processing CreateTablet for tablet aefb41ceeb524a28a6c17db6b11316e2 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c738d6165fed401e9f648ef1f8c6b97c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.374492 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefb41ceeb524a28a6c17db6b11316e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.376194 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 047bd48fda76496f820be523ad42dc5b (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c738d6165fed401e9f648ef1f8c6b97c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.376583 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047bd48fda76496f820be523ad42dc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.379819 20821 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:04.379958 23957 tablet_service.cc:1467] Processing CreateTablet for tablet a79db73c177047d8bab730a6c42637ba (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c738d6165fed401e9f648ef1f8c6b97c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.380383 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79db73c177047d8bab730a6c42637ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.383647 27285 consensus_queue.cc:1035] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.399081 27285 consensus_queue.cc:1035] T 4ed52ae966604421beb3ca1f381d99b3 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.430721 27734 tablet_bootstrap.cc:492] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:04.432868 27734 tablet_bootstrap.cc:654] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.442471 27696 tablet_bootstrap.cc:492] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:04.444411 27696 tablet_bootstrap.cc:654] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.467902 27696 tablet_bootstrap.cc:492] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:10:04.468183 27696 ts_tablet_manager.cc:1367] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.005s
I20240501 20:10:04.476955 27696 raft_consensus.cc:348] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.477398 27696 raft_consensus.cc:374] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.477586 27696 raft_consensus.cc:729] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.477939 27696 consensus_queue.cc:260] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.484516 27734 tablet_bootstrap.cc:492] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:10:04.484681 27734 ts_tablet_manager.cc:1367] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.007s
I20240501 20:10:04.486946 27769 tablet_bootstrap.cc:492] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:04.485374 27734 raft_consensus.cc:348] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:04.490360 27734 raft_consensus.cc:374] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.490442 27734 raft_consensus.cc:729] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.491215 27734 consensus_queue.cc:260] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:04.495842 27696 ts_tablet_manager.cc:1398] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240501 20:10:04.496248 27164 raft_consensus.cc:482] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.497367 27734 ts_tablet_manager.cc:1398] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:10:04.497311 27164 raft_consensus.cc:504] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.497691 27769 tablet_bootstrap.cc:654] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.498160 27164 leader_election.cc:288] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:04.500013 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefb41ceeb524a28a6c17db6b11316e2" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
W20240501 20:10:04.502424 21161 leader_election.cc:341] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:04.502763 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefb41ceeb524a28a6c17db6b11316e2" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
W20240501 20:10:04.503471 21162 leader_election.cc:341] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:04.503624 21162 leader_election.cc:302] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20; no voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:04.504223 27164 raft_consensus.cc:2726] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:04.506242 27763 tablet_bootstrap.cc:492] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:04.509544 27762 tablet_bootstrap.cc:492] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:04.513720 27763 tablet_bootstrap.cc:654] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.514051 27762 tablet_bootstrap.cc:654] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.524623 27763 tablet_bootstrap.cc:492] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:10:04.524840 27763 ts_tablet_manager.cc:1367] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 20:10:04.525547 27763 raft_consensus.cc:348] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.526221 27763 raft_consensus.cc:374] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.526335 27763 raft_consensus.cc:729] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.526573 27763 consensus_queue.cc:260] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.527143 27763 ts_tablet_manager.cc:1398] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:04.527874 27769 tablet_bootstrap.cc:492] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:04.528076 27769 ts_tablet_manager.cc:1367] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.008s
I20240501 20:10:04.528820 27763 tablet_bootstrap.cc:492] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:04.528995 27769 raft_consensus.cc:348] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:04.529954 27769 raft_consensus.cc:374] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.530334 27769 raft_consensus.cc:729] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.530539 27763 tablet_bootstrap.cc:654] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.530839 27769 consensus_queue.cc:260] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:04.531910 27762 tablet_bootstrap.cc:492] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:10:04.532116 27762 ts_tablet_manager.cc:1367] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240501 20:10:04.532794 27762 raft_consensus.cc:348] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.533104 27762 raft_consensus.cc:374] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.533224 27762 raft_consensus.cc:729] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.533489 27762 consensus_queue.cc:260] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.534731 27769 ts_tablet_manager.cc:1398] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:04.535658 27762 ts_tablet_manager.cc:1398] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:04.547744 27763 tablet_bootstrap.cc:492] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:10:04.547971 27763 ts_tablet_manager.cc:1367] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240501 20:10:04.548628 27763 raft_consensus.cc:348] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.548893 27763 raft_consensus.cc:374] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.549010 27763 raft_consensus.cc:729] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.549269 27763 consensus_queue.cc:260] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.549978 27763 ts_tablet_manager.cc:1398] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:04.550701 27763 tablet_bootstrap.cc:492] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:04.552403 27763 tablet_bootstrap.cc:654] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.568423 27763 tablet_bootstrap.cc:492] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159: No bootstrap required, opened a new log
I20240501 20:10:04.568769 27763 ts_tablet_manager.cc:1367] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240501 20:10:04.569820 27763 raft_consensus.cc:348] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:04.570140 27763 raft_consensus.cc:374] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.570271 27763 raft_consensus.cc:729] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.570600 27763 consensus_queue.cc:260] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:04.571276 27763 ts_tablet_manager.cc:1398] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:04.605540 27762 tablet_bootstrap.cc:492] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:04.607484 27762 tablet_bootstrap.cc:654] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.623929 27762 tablet_bootstrap.cc:492] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:10:04.624184 27762 ts_tablet_manager.cc:1367] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.010s
I20240501 20:10:04.625085 27762 raft_consensus.cc:348] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.625381 27762 raft_consensus.cc:374] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.625506 27762 raft_consensus.cc:729] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.625800 27762 consensus_queue.cc:260] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.626828 27762 ts_tablet_manager.cc:1398] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:04.628481 27518 raft_consensus.cc:482] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.628628 27518 raft_consensus.cc:504] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:04.629729 27518 leader_election.cc:288] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:04.629916 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047bd48fda76496f820be523ad42dc5b" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:04.630183 23710 raft_consensus.cc:2454] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:10:04.630548 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047bd48fda76496f820be523ad42dc5b" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:10:04.630749 21098 raft_consensus.cc:2454] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:10:04.630723 21293 leader_election.cc:302] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:10:04.631469 27518 raft_consensus.cc:2781] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:04.631636 27518 raft_consensus.cc:482] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:04.631734 27518 raft_consensus.cc:3037] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.636544 27518 raft_consensus.cc:504] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:04.638170 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047bd48fda76496f820be523ad42dc5b" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:10:04.638396 21098 raft_consensus.cc:3037] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.641502 21098 raft_consensus.cc:2454] T 047bd48fda76496f820be523ad42dc5b P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:10:04.642838 21291 leader_election.cc:302] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:04.643179 27345 raft_consensus.cc:2781] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:04.643291 27345 raft_consensus.cc:686] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:10:04.643550 27345 consensus_queue.cc:237] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:04.654982 27722 raft_consensus.cc:482] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.655205 27722 raft_consensus.cc:504] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.656209 27722 leader_election.cc:288] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:04.657003 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefb41ceeb524a28a6c17db6b11316e2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20" is_pre_election: true
I20240501 20:10:04.657232 21228 raft_consensus.cc:2454] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 0.
I20240501 20:10:04.658156 21294 leader_election.cc:302] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159; no voters:
I20240501 20:10:04.658313 21228 raft_consensus.cc:1261] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:04.659371 23843 raft_consensus.cc:1261] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:04.659621 27752 consensus_queue.cc:1035] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.661440 27722 raft_consensus.cc:2781] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:04.661550 27722 raft_consensus.cc:482] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:04.661623 27722 raft_consensus.cc:3037] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.661614 27767 tablet_bootstrap.cc:492] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:04.661792 27752 consensus_queue.cc:1035] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.664101 27722 raft_consensus.cc:504] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.664721 27722 leader_election.cc:288] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:04.667209 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefb41ceeb524a28a6c17db6b11316e2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:10:04.667490 20965 raft_consensus.cc:3037] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.668259 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefb41ceeb524a28a6c17db6b11316e2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:10:04.668560 21227 raft_consensus.cc:3037] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.669608 27518 leader_election.cc:288] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:04.670030 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefb41ceeb524a28a6c17db6b11316e2" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:10:04.673310 20965 raft_consensus.cc:2454] T aefb41ceeb524a28a6c17db6b11316e2 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:10:04.675746 21291 leader_election.cc:302] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:10:04.676479 27722 raft_consensus.cc:2781] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:04.676615 27722 raft_consensus.cc:686] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:10:04.676894 27722 consensus_queue.cc:237] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.692771 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047bd48fda76496f820be523ad42dc5b" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:10:04.693073 23710 raft_consensus.cc:3037] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.696017 23710 raft_consensus.cc:2454] T 047bd48fda76496f820be523ad42dc5b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:10:04.697113 21227 raft_consensus.cc:2454] T aefb41ceeb524a28a6c17db6b11316e2 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:10:04.699812 20821 catalog_manager.cc:5484] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:04.702564 27767 tablet_bootstrap.cc:654] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.710958 20818 catalog_manager.cc:5484] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:04.727286 27767 tablet_bootstrap.cc:492] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:04.729225 20821 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet 869370236097417a9942c450b2b02c40 (table rebalancer_dynamic_table [id=f2106faa2b9e4fb7b181571246c3df5b]). Expected version 1 got 0
I20240501 20:10:04.735868 27767 ts_tablet_manager.cc:1367] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20240501 20:10:04.736078 20818 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 869370236097417a9942c450b2b02c40 (table rebalancer_dynamic_table [id=f2106faa2b9e4fb7b181571246c3df5b]). Expected version 1 got 0
I20240501 20:10:04.740597 27783 tablet.cc:1722] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:04.741319 27783 tablet.cc:1620] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: MemRowSet was empty: no flush needed.
I20240501 20:10:04.739313 27767 raft_consensus.cc:348] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.742900 27767 raft_consensus.cc:374] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.743074 27767 raft_consensus.cc:729] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.742494 27785 tablet.cc:1722] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:04.743386 27767 consensus_queue.cc:260] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.744518 27767 ts_tablet_manager.cc:1398] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:10:04.747874 20824 catalog_manager.cc:5576] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349) does not have the latest schema for tablet 869370236097417a9942c450b2b02c40 (table rebalancer_dynamic_table [id=f2106faa2b9e4fb7b181571246c3df5b]). Expected version 1 got 0
I20240501 20:10:04.758293 27603 raft_consensus.cc:482] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.758560 27603 raft_consensus.cc:504] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.759501 27603 leader_election.cc:288] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:04.761185 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79db73c177047d8bab730a6c42637ba" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:10:04.761468 23975 raft_consensus.cc:2454] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:10:04.761942 20900 leader_election.cc:302] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:10:04.763211 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79db73c177047d8bab730a6c42637ba" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:10:04.763509 21357 raft_consensus.cc:2454] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 0.
I20240501 20:10:04.769577 27616 raft_consensus.cc:2781] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:04.769726 27616 raft_consensus.cc:482] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:04.769807 27616 raft_consensus.cc:3037] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.772804 27616 raft_consensus.cc:504] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.779242 27616 leader_election.cc:288] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:04.779942 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79db73c177047d8bab730a6c42637ba" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:10:04.780195 21357 raft_consensus.cc:3037] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.783015 21357 raft_consensus.cc:2454] T a79db73c177047d8bab730a6c42637ba P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:10:04.785151 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79db73c177047d8bab730a6c42637ba" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:10:04.785404 23975 raft_consensus.cc:3037] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.793321 27785 tablet.cc:1620] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:10:04.793542 23975 raft_consensus.cc:2454] T a79db73c177047d8bab730a6c42637ba P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:10:04.792307 20901 leader_election.cc:302] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:10:04.794332 27616 raft_consensus.cc:2781] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:04.794719 27616 raft_consensus.cc:686] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:10:04.795112 27616 consensus_queue.cc:237] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:04.803941 20818 catalog_manager.cc:5576] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431) does not have the latest schema for tablet 869370236097417a9942c450b2b02c40 (table rebalancer_dynamic_table [id=f2106faa2b9e4fb7b181571246c3df5b]). Expected version 1 got 0
I20240501 20:10:04.799685 27784 tablet.cc:1722] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:04.806664 27784 tablet.cc:1620] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:10:04.822082 20821 catalog_manager.cc:5484] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:04.823588 20824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=f2106faa2b9e4fb7b181571246c3df5b] alter complete (version 1)
I20240501 20:10:04.876797 21357 consensus_queue.cc:237] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
W20240501 20:10:04.879025 21293 consensus_peers.cc:485] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d14c3c71012405e81dabba5f127433d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:04.879362 21095 raft_consensus.cc:1261] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:04.879812 20964 raft_consensus.cc:1261] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:04.879923 27345 consensus_queue.cc:1035] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:04.880402 27345 consensus_queue.cc:1035] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:04.882952 27518 raft_consensus.cc:2932] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:04.883504 20964 raft_consensus.cc:2932] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:04.886438 21095 raft_consensus.cc:2932] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:04.888470 20818 catalog_manager.cc:5484] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:04.933032 23709 consensus_queue.cc:237] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:04.937003 21095 raft_consensus.cc:1261] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:04.937448 23646 consensus_peers.cc:485] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 719e9466ddd347e8810c3d631a770740. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:04.939966 27711 consensus_queue.cc:1035] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:04.945485 20821 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:04.961614 27783 tablet.cc:1722] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:04.962179 27783 tablet.cc:1620] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: MemRowSet was empty: no flush needed.
I20240501 20:10:04.962440 27784 tablet.cc:1722] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:04.962953 27784 tablet.cc:1620] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: MemRowSet was empty: no flush needed.
I20240501 20:10:04.963704 23842 raft_consensus.cc:1261] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:04.967212 27824 consensus_queue.cc:1035] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:05.007526 27785 tablet.cc:1722] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:05.008072 27785 tablet.cc:1620] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: MemRowSet was empty: no flush needed.
I20240501 20:10:05.010342 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table [id=f2106faa2b9e4fb7b181571246c3df5b] alter complete (version 2)
I20240501 20:10:05.023044 27711 raft_consensus.cc:2932] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:05.023867 21095 raft_consensus.cc:2932] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:05.028137 23842 raft_consensus.cc:2932] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:05.032847 20821 catalog_manager.cc:5484] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index -1 to 2, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:05.099843 27722 consensus_queue.cc:1035] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.106812 27345 consensus_queue.cc:1035] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.110095 27722 consensus_queue.cc:1035] T 047bd48fda76496f820be523ad42dc5b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.119421 20821 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:05.133802 27345 consensus_queue.cc:1035] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.147424 27783 tablet.cc:1722] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:05.148429 27784 tablet.cc:1722] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:05.159739 20821 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: rebalancer_dynamic_table_new_name [id=f2106faa2b9e4fb7b181571246c3df5b] alter complete (version 3)
I20240501 20:10:05.161965 27785 tablet.cc:1722] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:05.197389 27603 consensus_queue.cc:1035] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.242511 27616 consensus_queue.cc:1035] T a79db73c177047d8bab730a6c42637ba P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.263052 21095 consensus_queue.cc:237] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:05.266415 21228 raft_consensus.cc:1261] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:05.266399 20821 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:05.266574 20821 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43240:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240501 20:10:05.267271 21031 consensus_peers.cc:485] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Couldn't send request to peer 744c5a532e724391830c0785d32eb2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f392e77df7ff4da489b4d8a8dc415af0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:05.267941 20964 raft_consensus.cc:1261] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:05.270064 27856 consensus_queue.cc:1035] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:05.270792 20821 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f878915f05c04c2c8c7cd9cf39cd6cf8: Sending DeleteTablet for 3 replicas of tablet 869370236097417a9942c450b2b02c40
I20240501 20:10:05.270912 27856 consensus_queue.cc:1035] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:05.271915 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 869370236097417a9942c450b2b02c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:05 UTC) from {username='slave'} at 127.0.0.1:57076
I20240501 20:10:05.272465 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet 869370236097417a9942c450b2b02c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:05 UTC) from {username='slave'} at 127.0.0.1:45322
I20240501 20:10:05.272503 21076 tablet_service.cc:1514] Processing DeleteTablet for tablet 869370236097417a9942c450b2b02c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:05 UTC) from {username='slave'} at 127.0.0.1:51330
I20240501 20:10:05.284943 27858 tablet_replica.cc:330] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:10:05.289924 27859 tablet_replica.cc:330] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:10:05.290251 27859 raft_consensus.cc:2229] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:05.290702 27859 raft_consensus.cc:2258] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.292830 27859 ts_tablet_manager.cc:1875] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:05.294507 27754 raft_consensus.cc:2932] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:05.296180 27858 raft_consensus.cc:2229] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:05.296463 27858 raft_consensus.cc:2258] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.298128 20964 raft_consensus.cc:2932] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:05.303752 21228 raft_consensus.cc:2932] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:05.306169 20821 catalog_manager.cc:5484] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:05.310400 27859 ts_tablet_manager.cc:1888] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:05.310532 27859 log.cc:1196] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/869370236097417a9942c450b2b02c40
I20240501 20:10:05.310995 27859 ts_tablet_manager.cc:1909] T 869370236097417a9942c450b2b02c40 P feac73db0a3b4aafa374ca5dfd322b08: Deleting consensus metadata
I20240501 20:10:05.312091 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 869370236097417a9942c450b2b02c40 (table rebalancer_dynamic_table_new_name [id=f2106faa2b9e4fb7b181571246c3df5b]) successfully deleted
I20240501 20:10:05.333175 27858 ts_tablet_manager.cc:1875] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:05.330953 27857 tablet_replica.cc:330] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:10:05.336910 27857 raft_consensus.cc:2229] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:05.338886 27857 raft_consensus.cc:2258] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.340641 27857 ts_tablet_manager.cc:1875] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:05.343043 27862 ts_tablet_manager.cc:902] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:05.351619 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:05.353315 27858 ts_tablet_manager.cc:1888] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:05.355020 27857 ts_tablet_manager.cc:1888] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:05.355273 27857 log.cc:1196] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/869370236097417a9942c450b2b02c40
I20240501 20:10:05.357497 27857 ts_tablet_manager.cc:1909] T 869370236097417a9942c450b2b02c40 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting consensus metadata
I20240501 20:10:05.354512 27858 log.cc:1196] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/869370236097417a9942c450b2b02c40
I20240501 20:10:05.360169 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 869370236097417a9942c450b2b02c40 (table rebalancer_dynamic_table_new_name [id=f2106faa2b9e4fb7b181571246c3df5b]) successfully deleted
I20240501 20:10:05.361521 27858 ts_tablet_manager.cc:1909] T 869370236097417a9942c450b2b02c40 P 4af096f5a6a9440a94f6682b399efe20: Deleting consensus metadata
I20240501 20:10:05.363006 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 584178e395ec43169a4d14cbebde27e6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=72bf652e954042089c2060bb9796425e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.363165 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 584178e395ec43169a4d14cbebde27e6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=72bf652e954042089c2060bb9796425e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.363629 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584178e395ec43169a4d14cbebde27e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.364233 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584178e395ec43169a4d14cbebde27e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.364739 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 869370236097417a9942c450b2b02c40 (table rebalancer_dynamic_table_new_name [id=f2106faa2b9e4fb7b181571246c3df5b]) successfully deleted
I20240501 20:10:05.371912 24089 tablet_service.cc:1467] Processing CreateTablet for tablet 584178e395ec43169a4d14cbebde27e6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=72bf652e954042089c2060bb9796425e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.372328 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584178e395ec43169a4d14cbebde27e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.374446 23689 tablet_service.cc:1467] Processing CreateTablet for tablet d4ffd0f0fb874a809876af2a78ad07ab (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=72bf652e954042089c2060bb9796425e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.374985 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ffd0f0fb874a809876af2a78ad07ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.376459 20945 tablet_service.cc:1467] Processing CreateTablet for tablet c3aee98e618d453e9d02a80687682ada (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=72bf652e954042089c2060bb9796425e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.376853 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3aee98e618d453e9d02a80687682ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.376986 23957 tablet_service.cc:1467] Processing CreateTablet for tablet c3aee98e618d453e9d02a80687682ada (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=72bf652e954042089c2060bb9796425e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.377415 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3aee98e618d453e9d02a80687682ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.379252 21077 tablet_service.cc:1467] Processing CreateTablet for tablet c3aee98e618d453e9d02a80687682ada (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=72bf652e954042089c2060bb9796425e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.379647 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3aee98e618d453e9d02a80687682ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.380465 24088 tablet_service.cc:1467] Processing CreateTablet for tablet d4ffd0f0fb874a809876af2a78ad07ab (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=72bf652e954042089c2060bb9796425e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.380915 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ffd0f0fb874a809876af2a78ad07ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.383131 27871 tablet_bootstrap.cc:492] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:05.384902 20821 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43240:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:05.385164 27871 tablet_bootstrap.cc:654] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.394102 21207 tablet_service.cc:1467] Processing CreateTablet for tablet d4ffd0f0fb874a809876af2a78ad07ab (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=72bf652e954042089c2060bb9796425e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.394567 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ffd0f0fb874a809876af2a78ad07ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.397977 27871 tablet_bootstrap.cc:492] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:10:05.398432 27871 ts_tablet_manager.cc:1367] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240501 20:10:05.399442 27871 raft_consensus.cc:348] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.400902 27871 raft_consensus.cc:374] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.402565 27871 raft_consensus.cc:729] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.403159 27871 consensus_queue.cc:260] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.404476 27871 ts_tablet_manager.cc:1398] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:05.405380 27871 tablet_bootstrap.cc:492] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:05.406402 27873 tablet_bootstrap.cc:492] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:05.407310 27871 tablet_bootstrap.cc:654] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.409157 27875 tablet_bootstrap.cc:492] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:05.411402 27877 tablet_bootstrap.cc:492] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:05.412519 27875 tablet_bootstrap.cc:654] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.413408 27873 tablet_bootstrap.cc:654] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.414244 27877 tablet_bootstrap.cc:654] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.415427 27862 tablet_copy_client.cc:308] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:10:05.416177 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 4d14c3c71012405e81dabba5f127433d from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:37097)
I20240501 20:10:05.416344 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 4d14c3c71012405e81dabba5f127433d from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:37097: session id = 51bf640e895b4e5eb91bdf7509ea7e46-4d14c3c71012405e81dabba5f127433d
I20240501 20:10:05.420956 27876 ts_tablet_manager.cc:902] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:05.422674 21378 tablet_copy_source_session.cc:205] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:05.424880 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 33a6d66d523d4b44bf679fc11c651fdc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=17810ba68f084096956947373747ac78]), partition=
I20240501 20:10:05.425122 23956 tablet_service.cc:1467] Processing CreateTablet for tablet 33a6d66d523d4b44bf679fc11c651fdc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=17810ba68f084096956947373747ac78]), partition=
I20240501 20:10:05.425367 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a6d66d523d4b44bf679fc11c651fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.425549 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a6d66d523d4b44bf679fc11c651fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.426628 20945 tablet_service.cc:1467] Processing CreateTablet for tablet 33a6d66d523d4b44bf679fc11c651fdc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=17810ba68f084096956947373747ac78]), partition=
I20240501 20:10:05.426973 20945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a6d66d523d4b44bf679fc11c651fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.434298 27862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d14c3c71012405e81dabba5f127433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.437649 27871 tablet_bootstrap.cc:492] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08: No bootstrap required, opened a new log
I20240501 20:10:05.438078 27871 ts_tablet_manager.cc:1367] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.002s
I20240501 20:10:05.439364 27871 raft_consensus.cc:348] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:05.442389 27871 raft_consensus.cc:374] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.443174 27877 tablet_bootstrap.cc:492] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20: No bootstrap required, opened a new log
I20240501 20:10:05.443397 27877 ts_tablet_manager.cc:1367] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20240501 20:10:05.444398 27871 raft_consensus.cc:729] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.444417 27877 raft_consensus.cc:348] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.444732 27877 raft_consensus.cc:374] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.444844 27877 raft_consensus.cc:729] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.445195 27871 consensus_queue.cc:260] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:05.445225 27877 consensus_queue.cc:260] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.445809 27877 ts_tablet_manager.cc:1398] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:05.446643 27871 ts_tablet_manager.cc:1398] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 20:10:05.458151 27883 tablet_bootstrap.cc:492] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:05.460404 27886 tablet_bootstrap.cc:492] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:05.463310 27883 tablet_bootstrap.cc:654] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.464037 27886 tablet_bootstrap.cc:654] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.472887 27886 tablet_bootstrap.cc:492] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:10:05.474370 27886 ts_tablet_manager.cc:1367] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 20:10:05.476099 27873 tablet_bootstrap.cc:492] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:10:05.482620 27886 raft_consensus.cc:348] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.485080 27886 raft_consensus.cc:374] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.485484 27886 raft_consensus.cc:729] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.486284 27886 consensus_queue.cc:260] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.490835 27886 ts_tablet_manager.cc:1398] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 20:10:05.478309 27873 ts_tablet_manager.cc:1367] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.002s
I20240501 20:10:05.492974 27873 raft_consensus.cc:348] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:05.494954 27873 raft_consensus.cc:374] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.495126 27873 raft_consensus.cc:729] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.495460 27873 consensus_queue.cc:260] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:05.497562 27886 tablet_bootstrap.cc:492] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:05.499094 27873 ts_tablet_manager.cc:1398] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 20:10:05.503515 27886 tablet_bootstrap.cc:654] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.512815 27875 tablet_bootstrap.cc:492] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:05.513108 27875 ts_tablet_manager.cc:1367] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.104s user 0.005s sys 0.000s
I20240501 20:10:05.514299 27875 raft_consensus.cc:348] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.514647 27875 raft_consensus.cc:374] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.514802 27875 raft_consensus.cc:729] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.514957 27883 tablet_bootstrap.cc:492] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:05.515195 27883 ts_tablet_manager.cc:1367] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.000s
I20240501 20:10:05.515139 27875 consensus_queue.cc:260] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.515975 27875 ts_tablet_manager.cc:1398] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:05.516906 27883 raft_consensus.cc:348] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:05.517510 27883 raft_consensus.cc:374] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.517722 27883 raft_consensus.cc:729] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.518095 27883 consensus_queue.cc:260] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:05.518808 27285 raft_consensus.cc:482] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:05.519069 27873 tablet_bootstrap.cc:492] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:05.518970 27285 raft_consensus.cc:504] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.519843 27875 tablet_bootstrap.cc:492] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:05.519831 27285 leader_election.cc:288] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:05.520696 27883 ts_tablet_manager.cc:1398] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:10:05.520975 27883 tablet_bootstrap.cc:492] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:05.521602 27875 tablet_bootstrap.cc:654] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.522305 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584178e395ec43169a4d14cbebde27e6" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:05.522557 27883 tablet_bootstrap.cc:654] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.522579 23709 raft_consensus.cc:2454] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:10:05.524289 27862 tablet_copy_client.cc:783] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:05.524369 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584178e395ec43169a4d14cbebde27e6" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:10:05.524649 21095 raft_consensus.cc:2454] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 0.
I20240501 20:10:05.525209 27876 tablet_copy_client.cc:308] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.133:37155
I20240501 20:10:05.526099 24045 leader_election.cc:302] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:10:05.529381 27666 raft_consensus.cc:2781] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:05.529541 27666 raft_consensus.cc:482] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:05.529632 27666 raft_consensus.cc:3037] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.540926 27873 tablet_bootstrap.cc:654] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.546280 27862 tablet_copy_client.cc:647] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:05.546311 23734 tablet_copy_service.cc:140] P 51bf640e895b4e5eb91bdf7509ea7e46: Received BeginTabletCopySession request for tablet 719e9466ddd347e8810c3d631a770740 from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:60091)
I20240501 20:10:05.547394 23734 tablet_copy_service.cc:161] P 51bf640e895b4e5eb91bdf7509ea7e46: Beginning new tablet copy session on tablet 719e9466ddd347e8810c3d631a770740 from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:60091: session id = 8559229e044b4e1ea661b881b3f9af50-719e9466ddd347e8810c3d631a770740
I20240501 20:10:05.549433 27862 tablet_copy_client.cc:523] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:05.551050 23734 tablet_copy_source_session.cc:205] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:05.556869 27862 tablet_bootstrap.cc:492] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:05.559597 27666 raft_consensus.cc:504] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.560664 27666 leader_election.cc:288] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:05.561599 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584178e395ec43169a4d14cbebde27e6" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:10:05.561872 21095 raft_consensus.cc:3037] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.564987 21095 raft_consensus.cc:2454] T 584178e395ec43169a4d14cbebde27e6 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:10:05.567749 27873 tablet_bootstrap.cc:492] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0: No bootstrap required, opened a new log
I20240501 20:10:05.567920 27873 ts_tablet_manager.cc:1367] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20240501 20:10:05.570446 27873 raft_consensus.cc:348] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.572659 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584178e395ec43169a4d14cbebde27e6" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:10:05.572924 23710 raft_consensus.cc:3037] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.570392 24043 leader_election.cc:302] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:05.573587 27666 raft_consensus.cc:2781] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:05.573742 27666 raft_consensus.cc:686] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:10:05.572113 27876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719e9466ddd347e8810c3d631a770740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.574309 27666 consensus_queue.cc:237] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.580345 23710 raft_consensus.cc:2454] T 584178e395ec43169a4d14cbebde27e6 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:10:05.581264 27873 raft_consensus.cc:374] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.581508 27873 raft_consensus.cc:729] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.583976 27873 consensus_queue.cc:260] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.585404 27875 tablet_bootstrap.cc:492] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:05.585634 27875 ts_tablet_manager.cc:1367] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20240501 20:10:05.585760 27883 tablet_bootstrap.cc:492] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb: No bootstrap required, opened a new log
I20240501 20:10:05.586035 27883 ts_tablet_manager.cc:1367] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.007s
I20240501 20:10:05.586522 27875 raft_consensus.cc:348] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.586777 27875 raft_consensus.cc:374] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.586941 27875 raft_consensus.cc:729] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.587044 27883 raft_consensus.cc:348] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.587317 27875 consensus_queue.cc:260] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.587455 27883 raft_consensus.cc:374] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.587571 27883 raft_consensus.cc:729] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.587888 27883 consensus_queue.cc:260] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.589335 27875 ts_tablet_manager.cc:1398] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:05.589697 27875 tablet_bootstrap.cc:492] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:05.591017 27883 ts_tablet_manager.cc:1398] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:05.594110 27886 tablet_bootstrap.cc:492] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50: No bootstrap required, opened a new log
I20240501 20:10:05.594904 27886 ts_tablet_manager.cc:1367] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.102s user 0.001s sys 0.009s
I20240501 20:10:05.596052 27886 raft_consensus.cc:348] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.596428 27886 raft_consensus.cc:374] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.596556 27886 raft_consensus.cc:729] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.597111 27886 consensus_queue.cc:260] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.597774 27886 ts_tablet_manager.cc:1398] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:05.597966 27873 ts_tablet_manager.cc:1398] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20240501 20:10:05.599486 27876 tablet_copy_client.cc:783] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:05.599854 27876 tablet_copy_client.cc:647] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:05.607468 27875 tablet_bootstrap.cc:654] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.610075 20824 catalog_manager.cc:5484] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:05.611141 27164 raft_consensus.cc:482] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:05.611318 27164 raft_consensus.cc:504] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.612015 27876 tablet_copy_client.cc:523] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:05.612175 27164 leader_election.cc:288] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:05.612896 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:10:05.613159 24108 raft_consensus.cc:2454] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:10:05.613648 21163 leader_election.cc:302] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:10:05.614208 27164 raft_consensus.cc:2781] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:05.614310 27164 raft_consensus.cc:482] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:05.614291 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:10:05.614418 27164 raft_consensus.cc:3037] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.614552 23710 raft_consensus.cc:2454] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 0.
I20240501 20:10:05.618820 27876 tablet_bootstrap.cc:492] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:05.626338 27164 raft_consensus.cc:504] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.628085 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:10:05.628392 23710 raft_consensus.cc:3037] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.630501 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "4af096f5a6a9440a94f6682b399efe20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:10:05.630749 24108 raft_consensus.cc:3037] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.631982 27836 raft_consensus.cc:482] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:05.632135 27836 raft_consensus.cc:504] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:05.632421 23710 raft_consensus.cc:2454] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:10:05.633095 27836 leader_election.cc:288] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:05.633899 24108 raft_consensus.cc:2454] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af096f5a6a9440a94f6682b399efe20 in term 1.
I20240501 20:10:05.634477 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3aee98e618d453e9d02a80687682ada" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:10:05.634716 21097 raft_consensus.cc:2454] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:10:05.635216 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3aee98e618d453e9d02a80687682ada" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:10:05.635437 20964 raft_consensus.cc:2454] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 0.
I20240501 20:10:05.636188 23909 leader_election.cc:302] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:10:05.636870 27650 raft_consensus.cc:2781] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:05.636979 27650 raft_consensus.cc:482] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:05.637102 27650 raft_consensus.cc:3037] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.640715 27650 raft_consensus.cc:504] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:05.641582 27650 leader_election.cc:288] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:05.642606 21095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3aee98e618d453e9d02a80687682ada" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:10:05.642827 21095 raft_consensus.cc:3037] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.627306 27164 leader_election.cc:288] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:05.633432 21163 leader_election.cc:302] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46; no voters:
I20240501 20:10:05.644910 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3aee98e618d453e9d02a80687682ada" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:10:05.645144 20964 raft_consensus.cc:3037] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.650276 27875 tablet_bootstrap.cc:492] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46: No bootstrap required, opened a new log
I20240501 20:10:05.650465 27875 ts_tablet_manager.cc:1367] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20240501 20:10:05.650718 21095 raft_consensus.cc:2454] T c3aee98e618d453e9d02a80687682ada P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:10:05.651536 27875 raft_consensus.cc:348] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.651917 27875 raft_consensus.cc:374] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.652035 27875 raft_consensus.cc:729] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.652426 27875 consensus_queue.cc:260] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.653347 20964 raft_consensus.cc:2454] T c3aee98e618d453e9d02a80687682ada P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 1.
I20240501 20:10:05.652730 23909 leader_election.cc:302] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:05.654016 27164 raft_consensus.cc:2781] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:05.654245 27650 raft_consensus.cc:2781] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:05.654446 27650 raft_consensus.cc:686] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:10:05.654937 27650 consensus_queue.cc:237] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:05.656162 27164 raft_consensus.cc:686] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Leader. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:10:05.656471 27164 consensus_queue.cc:237] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.657882 27875 ts_tablet_manager.cc:1398] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240501 20:10:05.665374 27876 tablet_bootstrap.cc:492] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:05.667151 27876 tablet_bootstrap.cc:492] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:10:05.667686 20821 catalog_manager.cc:5484] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:05.667975 27876 ts_tablet_manager.cc:1367] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.001s
I20240501 20:10:05.672499 27876 raft_consensus.cc:348] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.674120 27876 raft_consensus.cc:729] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.674739 27876 consensus_queue.cc:260] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:05.676151 27876 ts_tablet_manager.cc:1398] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 20:10:05.678586 23734 tablet_copy_service.cc:342] P 51bf640e895b4e5eb91bdf7509ea7e46: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-719e9466ddd347e8810c3d631a770740 received from {username='slave'} at 127.13.251.136:60091
I20240501 20:10:05.678737 23734 tablet_copy_service.cc:433] P 51bf640e895b4e5eb91bdf7509ea7e46: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-719e9466ddd347e8810c3d631a770740 on tablet 719e9466ddd347e8810c3d631a770740 with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:10:05.696158 20824 catalog_manager.cc:5484] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131). New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:05.734642 23977 consensus_queue.cc:237] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
W20240501 20:10:05.734817 21013 tablet.cc:2369] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:10:05.749541 23909 consensus_peers.cc:485] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8ef72f9e1d04bb498f540a74a850ed8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:05.749526 21095 raft_consensus.cc:1261] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:05.750030 23710 raft_consensus.cc:1261] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:05.750183 27650 consensus_queue.cc:1035] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:05.751844 27650 consensus_queue.cc:1035] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:05.757402 27650 raft_consensus.cc:2932] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:05.760550 23709 raft_consensus.cc:2932] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:05.770426 20824 catalog_manager.cc:5484] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:05.774979 27862 tablet_bootstrap.cc:492] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:05.775486 27862 tablet_bootstrap.cc:492] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:10:05.775853 27862 ts_tablet_manager.cc:1367] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.219s user 0.052s sys 0.003s
I20240501 20:10:05.776705 27862 raft_consensus.cc:348] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.777019 27862 raft_consensus.cc:729] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.777304 27862 consensus_queue.cc:260] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.779358 27862 ts_tablet_manager.cc:1398] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:05.779966 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-4d14c3c71012405e81dabba5f127433d received from {username='slave'} at 127.13.251.133:37097
I20240501 20:10:05.780066 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-4d14c3c71012405e81dabba5f127433d on tablet 4d14c3c71012405e81dabba5f127433d with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:10:05.784080 21095 raft_consensus.cc:2932] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:05.795078 24108 raft_consensus.cc:1203] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:05.800058 27919 ts_tablet_manager.cc:902] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
W20240501 20:10:05.817405 24023 tablet.cc:2369] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:05.819662 27919 tablet_copy_client.cc:308] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:10:05.821760 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet f392e77df7ff4da489b4d8a8dc415af0 from peer 744c5a532e724391830c0785d32eb2fb ({username='slave'} at 127.13.251.135:53413)
I20240501 20:10:05.821949 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet f392e77df7ff4da489b4d8a8dc415af0 from peer 744c5a532e724391830c0785d32eb2fb at {username='slave'} at 127.13.251.135:53413: session id = 744c5a532e724391830c0785d32eb2fb-f392e77df7ff4da489b4d8a8dc415af0
I20240501 20:10:05.825387 21118 tablet_copy_source_session.cc:205] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:05.826763 27919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f392e77df7ff4da489b4d8a8dc415af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.832882 27919 tablet_copy_client.cc:783] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:05.833169 27919 tablet_copy_client.cc:647] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:05.834988 27919 tablet_copy_client.cc:523] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:05.838594 27919 tablet_bootstrap.cc:492] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:05.884738 27919 tablet_bootstrap.cc:492] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:05.885327 27919 tablet_bootstrap.cc:492] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: Bootstrap complete.
I20240501 20:10:05.885710 27919 ts_tablet_manager.cc:1367] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.047s user 0.049s sys 0.000s
I20240501 20:10:05.886518 27919 raft_consensus.cc:348] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:05.886821 27919 raft_consensus.cc:729] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.887097 27919 consensus_queue.cc:260] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:05.887768 27919 ts_tablet_manager.cc:1398] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:05.890089 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session 744c5a532e724391830c0785d32eb2fb-f392e77df7ff4da489b4d8a8dc415af0 received from {username='slave'} at 127.13.251.135:53413
I20240501 20:10:05.890233 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session 744c5a532e724391830c0785d32eb2fb-f392e77df7ff4da489b4d8a8dc415af0 on tablet f392e77df7ff4da489b4d8a8dc415af0 with peer 744c5a532e724391830c0785d32eb2fb
I20240501 20:10:05.930491 27823 raft_consensus.cc:482] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:05.930693 27823 raft_consensus.cc:504] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.931339 27823 leader_election.cc:288] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:05.934216 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a6d66d523d4b44bf679fc11c651fdc" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:10:05.934263 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a6d66d523d4b44bf679fc11c651fdc" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:10:05.934513 23977 raft_consensus.cc:2454] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:10:05.934509 20964 raft_consensus.cc:2454] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 0.
I20240501 20:10:05.935106 23644 leader_election.cc:302] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:10:05.938416 27600 raft_consensus.cc:2781] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:05.938556 27600 raft_consensus.cc:482] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:05.938647 27600 raft_consensus.cc:3037] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.943303 27600 raft_consensus.cc:504] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.944777 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a6d66d523d4b44bf679fc11c651fdc" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:10:05.945039 23977 raft_consensus.cc:3037] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.950210 23977 raft_consensus.cc:2454] T 33a6d66d523d4b44bf679fc11c651fdc P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:10:05.951370 23644 leader_election.cc:302] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:10:05.951752 27823 raft_consensus.cc:2781] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:05.954404 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a6d66d523d4b44bf679fc11c651fdc" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:10:05.954615 20964 raft_consensus.cc:3037] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.957377 20964 raft_consensus.cc:2454] T 33a6d66d523d4b44bf679fc11c651fdc P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:10:05.958804 27600 leader_election.cc:288] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:05.959815 27823 raft_consensus.cc:686] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:10:05.960107 27823 consensus_queue.cc:237] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:05.993827 20824 catalog_manager.cc:5484] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:06.063725 27666 consensus_queue.cc:1035] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.074064 27650 consensus_queue.cc:1035] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.114122 27900 consensus_queue.cc:1035] T 584178e395ec43169a4d14cbebde27e6 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.131110 27836 consensus_queue.cc:1035] T c3aee98e618d453e9d02a80687682ada P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.144551 27164 consensus_queue.cc:1035] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.167898 26629 consensus_queue.cc:1035] T d4ffd0f0fb874a809876af2a78ad07ab P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 20:10:06.169843 27978 ts_tablet_manager.cc:902] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:06.209975 27978 tablet_copy_client.cc:308] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:10:06.211442 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet d8ef72f9e1d04bb498f540a74a850ed8 from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:58569)
I20240501 20:10:06.211663 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet d8ef72f9e1d04bb498f540a74a850ed8 from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:58569: session id = cda19d6a81c0473db302535ce2ada7f0-d8ef72f9e1d04bb498f540a74a850ed8
I20240501 20:10:06.244531 23710 raft_consensus.cc:1203] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:06.264174 23997 tablet_copy_source_session.cc:205] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:06.268687 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ef72f9e1d04bb498f540a74a850ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.274758 27978 tablet_copy_client.cc:783] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:06.275024 27978 tablet_copy_client.cc:647] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:06.277622 27978 tablet_copy_client.cc:523] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:06.280894 27978 tablet_bootstrap.cc:492] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:06.293265 27978 tablet_bootstrap.cc:492] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:06.293759 27978 tablet_bootstrap.cc:492] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:10:06.294095 27978 ts_tablet_manager.cc:1367] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.005s
I20240501 20:10:06.295264 27978 raft_consensus.cc:348] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:06.295631 27978 raft_consensus.cc:729] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.295984 27978 consensus_queue.cc:260] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:06.296648 27978 ts_tablet_manager.cc:1398] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:06.299960 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-d8ef72f9e1d04bb498f540a74a850ed8 received from {username='slave'} at 127.13.251.129:58569
I20240501 20:10:06.300101 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-d8ef72f9e1d04bb498f540a74a850ed8 on tablet d8ef72f9e1d04bb498f540a74a850ed8 with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:06.342551 23977 raft_consensus.cc:1203] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:06.460075 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:06.460276 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:06.504953 27958 consensus_queue.cc:1035] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.517339 27958 consensus_queue.cc:1035] T 33a6d66d523d4b44bf679fc11c651fdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.518383 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:06.538519 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:06.551816 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:06.576720 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:06.594900 20967 raft_consensus.cc:1203] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:06.605674 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:06.605907 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 20:10:07.037329 21161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 60824.8us
I20240501 20:10:07.155416 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:07.156994 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:07.213733 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:07.220007 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:07.239849 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:07.312747 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:07.337380 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:07.377820 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:07.564232 21355 consensus_queue.cc:237] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:07.566517 21098 raft_consensus.cc:1261] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.566562 23712 raft_consensus.cc:1261] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.567034 28098 consensus_queue.cc:1035] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:07.567332 28085 consensus_queue.cc:1035] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:07.570618 28085 raft_consensus.cc:2932] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:07.574033 23712 raft_consensus.cc:2932] T 4d14c3c71012405e81dabba5f127433d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:07.574703 21098 raft_consensus.cc:2932] T 4d14c3c71012405e81dabba5f127433d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:07.577065 20826 catalog_manager.cc:5484] T 4d14c3c71012405e81dabba5f127433d P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:07.625660 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d14c3c71012405e81dabba5f127433d with delete_type TABLET_DATA_TOMBSTONED (TS cda19d6a81c0473db302535ce2ada7f0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54238
I20240501 20:10:07.626266 28127 tablet_replica.cc:330] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:10:07.626575 28127 raft_consensus.cc:2229] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:07.626859 28127 raft_consensus.cc:2258] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:07.627646 28127 ts_tablet_manager.cc:1875] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:07.633344 28127 ts_tablet_manager.cc:1888] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:07.633452 28127 log.cc:1196] T 4d14c3c71012405e81dabba5f127433d P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/4d14c3c71012405e81dabba5f127433d
I20240501 20:10:07.634241 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 4d14c3c71012405e81dabba5f127433d (table rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]) successfully deleted
I20240501 20:10:07.862699 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:07.877894 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:07.890496 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:07.907119 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:07.919557 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:08.004918 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:08.011801 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:10:08.029337 24043 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 48913.2us
I20240501 20:10:08.053557 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:08.206246 21227 consensus_queue.cc:237] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:08.208830 23846 raft_consensus.cc:1261] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.208983 21355 raft_consensus.cc:1261] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.209810 26629 consensus_queue.cc:1035] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.210100 27164 consensus_queue.cc:1035] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.212998 28073 raft_consensus.cc:2932] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:08.213598 21355 raft_consensus.cc:2932] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:08.219285 20821 catalog_manager.cc:5484] T 420b46edf5fc438fb01e822440bce8f1 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:08.242872 23842 raft_consensus.cc:2932] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:08.246488 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet 420b46edf5fc438fb01e822440bce8f1 with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51330
I20240501 20:10:08.247116 28206 tablet_replica.cc:330] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:10:08.248721 28206 raft_consensus.cc:2229] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.249063 28206 raft_consensus.cc:2258] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.249655 28206 ts_tablet_manager.cc:1875] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.255021 28206 ts_tablet_manager.cc:1888] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:08.255198 28206 log.cc:1196] T 420b46edf5fc438fb01e822440bce8f1 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/420b46edf5fc438fb01e822440bce8f1
I20240501 20:10:08.256040 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 420b46edf5fc438fb01e822440bce8f1 (table rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]) successfully deleted
I20240501 20:10:08.517154 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:08.555389 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:08.632969 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:08.656395 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:08.678534 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:08.687721 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:08.696425 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:08.748647 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:08.848486 24109 consensus_queue.cc:237] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:08.852319 23712 raft_consensus.cc:1261] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:08.852330 21095 raft_consensus.cc:1261] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:08.852988 28182 consensus_queue.cc:1035] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:08.853253 27666 consensus_queue.cc:1035] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:08.855703 28182 raft_consensus.cc:2932] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:08.856494 21098 raft_consensus.cc:2932] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:08.860247 23712 raft_consensus.cc:2932] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:08.864636 20826 catalog_manager.cc:5484] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 2 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 2 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:08.883379 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 30eb2ed6c8b64204b4c3a11b41e53ec8 with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:08.892912 28310 tablet_replica.cc:330] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:08.893144 28310 raft_consensus.cc:2229] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.893381 28310 raft_consensus.cc:2258] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.894003 28310 ts_tablet_manager.cc:1875] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.900179 28310 ts_tablet_manager.cc:1888] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 20:10:08.900323 28310 log.cc:1196] T 30eb2ed6c8b64204b4c3a11b41e53ec8 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/30eb2ed6c8b64204b4c3a11b41e53ec8
I20240501 20:10:08.901108 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 30eb2ed6c8b64204b4c3a11b41e53ec8 (table rebalancer_extra_table_4 [id=2e61e0dca7d44277902734a19c35aed8]) successfully deleted
I20240501 20:10:09.235626 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:09.241348 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:09.273588 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:09.299042 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:09.334199 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:09.389057 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:09.393564 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:09.408867 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:09.590873 21227 consensus_queue.cc:237] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:09.595257 23975 raft_consensus.cc:1261] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:09.595973 20967 raft_consensus.cc:1261] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:09.596220 28284 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:09.596606 26629 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:09.600731 26629 raft_consensus.cc:2932] T 30d2aada073b4d8989d7db8be4034756 P 4af096f5a6a9440a94f6682b399efe20 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:09.605226 23977 raft_consensus.cc:2932] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:09.606323 20967 raft_consensus.cc:2932] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:09.613754 20820 catalog_manager.cc:5484] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 2 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:09.630748 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d2aada073b4d8989d7db8be4034756 with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42940
I20240501 20:10:09.636898 28400 tablet_replica.cc:330] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:10:09.637117 28400 raft_consensus.cc:2229] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:09.637293 28400 raft_consensus.cc:2258] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:09.638283 28400 ts_tablet_manager.cc:1875] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:09.643826 28400 ts_tablet_manager.cc:1888] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:10:09.643939 28400 log.cc:1196] T 30d2aada073b4d8989d7db8be4034756 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/30d2aada073b4d8989d7db8be4034756
I20240501 20:10:09.644819 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 30d2aada073b4d8989d7db8be4034756 (table rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]) successfully deleted
I20240501 20:10:09.970288 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:09.999298 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:10.127349 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:10.168612 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:10.175845 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:10.197038 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:10.198657 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:10.198956 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:10.392184 21095 consensus_queue.cc:237] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:10.394454 23977 raft_consensus.cc:1261] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:10.394497 20963 raft_consensus.cc:1261] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:10.395056 27754 consensus_queue.cc:1035] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:10.395622 27752 consensus_queue.cc:1035] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:10.398311 27752 raft_consensus.cc:2932] T f392e77df7ff4da489b4d8a8dc415af0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:10.398830 20967 raft_consensus.cc:2932] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:10.403788 23977 raft_consensus.cc:2932] T f392e77df7ff4da489b4d8a8dc415af0 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:10.404846 20821 catalog_manager.cc:5484] T f392e77df7ff4da489b4d8a8dc415af0 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:10.411331 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet f392e77df7ff4da489b4d8a8dc415af0 with delete_type TABLET_DATA_TOMBSTONED (TS 4af096f5a6a9440a94f6682b399efe20 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45322
I20240501 20:10:10.412679 28498 tablet_replica.cc:330] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:10:10.412915 28498 raft_consensus.cc:2229] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:10.413305 28498 raft_consensus.cc:2258] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:10.414098 28498 ts_tablet_manager.cc:1875] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:10.419178 28498 ts_tablet_manager.cc:1888] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:10.419288 28498 log.cc:1196] T f392e77df7ff4da489b4d8a8dc415af0 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/f392e77df7ff4da489b4d8a8dc415af0
I20240501 20:10:10.419974 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet f392e77df7ff4da489b4d8a8dc415af0 (table rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]) successfully deleted
I20240501 20:10:10.786851 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:10.789173 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:10.825762 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:10.838272 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:10.872905 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:10.879122 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:10.960834 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:10.992178 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:11.147336 23709 consensus_queue.cc:237] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:11.149875 21095 raft_consensus.cc:1261] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:11.149911 24110 raft_consensus.cc:1261] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:11.150518 28096 consensus_queue.cc:1035] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:11.151006 28096 consensus_queue.cc:1035] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:11.155148 28472 raft_consensus.cc:2932] T 719e9466ddd347e8810c3d631a770740 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.156069 24110 raft_consensus.cc:2932] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.162668 20818 catalog_manager.cc:5484] T 719e9466ddd347e8810c3d631a770740 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.173780 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 719e9466ddd347e8810c3d631a770740 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:10:11.174571 28612 tablet_replica.cc:330] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:10:11.174922 28612 raft_consensus.cc:2229] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:11.175415 28612 raft_consensus.cc:2258] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:11.175161 21095 raft_consensus.cc:2932] T 719e9466ddd347e8810c3d631a770740 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.176184 28612 ts_tablet_manager.cc:1875] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:11.183593 28612 ts_tablet_manager.cc:1888] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:11.183781 28612 log.cc:1196] T 719e9466ddd347e8810c3d631a770740 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/719e9466ddd347e8810c3d631a770740
I20240501 20:10:11.185295 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 719e9466ddd347e8810c3d631a770740 (table rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]) successfully deleted
I20240501 20:10:11.215135 20963 consensus_queue.cc:237] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:11.217494 21095 raft_consensus.cc:1261] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:11.218334 21357 raft_consensus.cc:1261] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:11.218924 20902 consensus_peers.cc:485] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75fa04c8726c4f2e932740cc6fbdcdd0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.219146 28584 consensus_queue.cc:1035] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:11.219650 28459 consensus_queue.cc:1035] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:11.237980 28459 raft_consensus.cc:2932] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.240444 21095 raft_consensus.cc:2932] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.251459 20818 catalog_manager.cc:5484] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.247033 21357 raft_consensus.cc:2932] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.293716 23842 consensus_queue.cc:237] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:11.296629 21227 raft_consensus.cc:1261] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:11.296679 23779 consensus_peers.cc:485] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad72fa9eb71c4a2f9eaee8f48f7bbca9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.296691 24110 raft_consensus.cc:1261] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.297242 27847 consensus_queue.cc:1035] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.297547 28649 consensus_queue.cc:1035] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.299954 28649 raft_consensus.cc:2932] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:11.300626 21227 raft_consensus.cc:2932] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:11.306187 24110 raft_consensus.cc:2932] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:11.306888 20819 catalog_manager.cc:5484] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:11.352062 23709 consensus_queue.cc:237] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:11.354912 24110 raft_consensus.cc:1261] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.354998 23842 raft_consensus.cc:1261] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.355477 28096 consensus_queue.cc:1035] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:11.355778 23644 consensus_peers.cc:485] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb612facbc174e0b957294af03ef2864. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.355782 28472 consensus_queue.cc:1035] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.358842 28472 raft_consensus.cc:2932] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:11.361019 24109 raft_consensus.cc:2932] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:11.364137 23842 raft_consensus.cc:2932] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:11.366503 20820 catalog_manager.cc:5484] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:11.410199 23975 consensus_queue.cc:237] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:11.412741 20963 raft_consensus.cc:1261] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.412755 23709 raft_consensus.cc:1261] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.413434 28473 consensus_queue.cc:1035] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:11.413695 23910 consensus_peers.cc:485] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 182bd3935cce49868a451685097e9f0c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.413820 28501 consensus_queue.cc:1035] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.428853 28501 raft_consensus.cc:2932] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:11.429481 23709 raft_consensus.cc:2932] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:11.432791 20963 raft_consensus.cc:2932] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:11.437454 20820 catalog_manager.cc:5484] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:11.490417 21095 consensus_queue.cc:237] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
W20240501 20:10:11.496898 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1be0c0baa5e4c95915b93f659c4759f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.497233 24109 raft_consensus.cc:1261] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.497750 28493 consensus_queue.cc:1035] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.498390 23842 raft_consensus.cc:1261] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.499006 28493 consensus_queue.cc:1035] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.515599 28493 raft_consensus.cc:2932] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:11.522269 23842 raft_consensus.cc:2932] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:11.522323 24109 raft_consensus.cc:2932] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:11.532233 20820 catalog_manager.cc:5484] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:11.607661 28723 ts_tablet_manager.cc:902] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:11.616343 23842 consensus_queue.cc:237] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:11.618933 23975 raft_consensus.cc:1261] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:11.618999 23779 consensus_peers.cc:485] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d37fde63d6414745b0f81c0a0abe4f46. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.619395 28723 tablet_copy_client.cc:308] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:41259
I20240501 20:10:11.619560 24109 raft_consensus.cc:1261] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.619712 28389 consensus_queue.cc:1035] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.620096 20987 tablet_copy_service.cc:140] P cda19d6a81c0473db302535ce2ada7f0: Received BeginTabletCopySession request for tablet 75fa04c8726c4f2e932740cc6fbdcdd0 from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:35157)
I20240501 20:10:11.620121 28649 consensus_queue.cc:1035] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.620255 20987 tablet_copy_service.cc:161] P cda19d6a81c0473db302535ce2ada7f0: Beginning new tablet copy session on tablet 75fa04c8726c4f2e932740cc6fbdcdd0 from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:35157: session id = 8559229e044b4e1ea661b881b3f9af50-75fa04c8726c4f2e932740cc6fbdcdd0
I20240501 20:10:11.623344 20987 tablet_copy_source_session.cc:205] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:11.626605 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75fa04c8726c4f2e932740cc6fbdcdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.635221 28723 tablet_copy_client.cc:783] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:11.635437 28723 tablet_copy_client.cc:647] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:11.637216 28723 tablet_copy_client.cc:523] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:11.640480 28723 tablet_bootstrap.cc:492] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:11.641075 28389 raft_consensus.cc:2932] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:11.641599 23975 raft_consensus.cc:2932] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:11.647353 24109 raft_consensus.cc:2932] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:11.648788 20820 catalog_manager.cc:5484] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:11.707473 23842 consensus_queue.cc:237] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:11.709888 21227 raft_consensus.cc:1261] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.711063 28649 consensus_queue.cc:1035] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.711202 28723 tablet_bootstrap.cc:492] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240501 20:10:11.711347 23779 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c58dd83ebd0949dea6f4567067c8dfc9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.711577 28723 tablet_bootstrap.cc:492] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:10:11.711822 28723 ts_tablet_manager.cc:1367] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.071s user 0.047s sys 0.021s
I20240501 20:10:11.712677 23709 raft_consensus.cc:1261] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.712739 28723 raft_consensus.cc:348] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:11.712953 28723 raft_consensus.cc:729] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.713307 28649 consensus_queue.cc:1035] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.713201 28723 consensus_queue.cc:260] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:11.714722 28723 ts_tablet_manager.cc:1398] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:10:11.715390 20987 tablet_copy_service.cc:342] P cda19d6a81c0473db302535ce2ada7f0: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-75fa04c8726c4f2e932740cc6fbdcdd0 received from {username='slave'} at 127.13.251.136:35157
I20240501 20:10:11.715504 20987 tablet_copy_service.cc:433] P cda19d6a81c0473db302535ce2ada7f0: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-75fa04c8726c4f2e932740cc6fbdcdd0 on tablet 75fa04c8726c4f2e932740cc6fbdcdd0 with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:10:11.717157 28649 raft_consensus.cc:2932] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:11.717640 23709 raft_consensus.cc:2932] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:11.724547 21228 raft_consensus.cc:2932] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:11.727093 20819 catalog_manager.cc:5484] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:11.773641 21095 consensus_queue.cc:237] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:11.776016 23709 raft_consensus.cc:1261] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:11.776831 21032 consensus_peers.cc:485] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ffdb911cf1547329552d2e6af92c6df. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.777310 28493 consensus_queue.cc:1035] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.778890 23977 raft_consensus.cc:1261] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.779947 28493 consensus_queue.cc:1035] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.783246 28710 raft_consensus.cc:2932] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:11.783787 23975 raft_consensus.cc:2932] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:11.788671 23712 raft_consensus.cc:2932] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:11.791373 20818 catalog_manager.cc:5484] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:11.841310 21357 consensus_queue.cc:237] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:11.843992 21228 raft_consensus.cc:1261] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:11.844151 21095 raft_consensus.cc:1261] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:11.844481 21293 consensus_peers.cc:485] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72726d6b786e4c72ace7359101cec949. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.844805 28366 consensus_queue.cc:1035] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:11.845088 28098 consensus_queue.cc:1035] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:11.851992 28792 ts_tablet_manager.cc:902] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:11.859023 28366 raft_consensus.cc:2932] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.859618 21095 raft_consensus.cc:2932] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.866044 20820 catalog_manager.cc:5484] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.867509 28792 tablet_copy_client.cc:308] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:10:11.868377 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9 from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:45053)
I20240501 20:10:11.868568 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9 from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:45053: session id = cda19d6a81c0473db302535ce2ada7f0-ad72fa9eb71c4a2f9eaee8f48f7bbca9
I20240501 20:10:11.872814 21228 raft_consensus.cc:2932] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:11.879659 23866 tablet_copy_source_session.cc:205] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:11.881121 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.887634 28802 ts_tablet_manager.cc:902] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:10:11.888233 28792 tablet_copy_client.cc:783] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:11.888890 28792 tablet_copy_client.cc:647] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:11.892030 28792 tablet_copy_client.cc:523] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:11.892061 28802 tablet_copy_client.cc:308] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:10:11.892907 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet e1be0c0baa5e4c95915b93f659c4759f from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:45933)
I20240501 20:10:11.893057 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet e1be0c0baa5e4c95915b93f659c4759f from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:45933: session id = 4af096f5a6a9440a94f6682b399efe20-e1be0c0baa5e4c95915b93f659c4759f
I20240501 20:10:11.895121 28792 tablet_bootstrap.cc:492] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:11.895830 21118 tablet_copy_source_session.cc:205] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:11.897085 28802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1be0c0baa5e4c95915b93f659c4759f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.901124 28792 tablet_bootstrap.cc:492] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:11.901765 28792 tablet_bootstrap.cc:492] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:10:11.902021 28792 ts_tablet_manager.cc:1367] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:10:11.902628 28802 tablet_copy_client.cc:783] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:11.902750 28792 raft_consensus.cc:348] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:11.902915 28802 tablet_copy_client.cc:647] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:11.902966 28792 raft_consensus.cc:729] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.903172 28792 consensus_queue.cc:260] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:11.903738 28792 ts_tablet_manager.cc:1398] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:11.904676 28802 tablet_copy_client.cc:523] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:11.905061 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-ad72fa9eb71c4a2f9eaee8f48f7bbca9 received from {username='slave'} at 127.13.251.129:45053
I20240501 20:10:11.905212 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-ad72fa9eb71c4a2f9eaee8f48f7bbca9 on tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9 with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:11.910995 28802 tablet_bootstrap.cc:492] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:11.926038 28802 tablet_bootstrap.cc:492] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:11.926517 28802 tablet_bootstrap.cc:492] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:10:11.926792 28802 ts_tablet_manager.cc:1367] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.005s
I20240501 20:10:11.927448 28802 raft_consensus.cc:348] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:11.927745 28802 raft_consensus.cc:729] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.928134 28802 consensus_queue.cc:260] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:11.928787 28802 ts_tablet_manager.cc:1398] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:11.929508 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-e1be0c0baa5e4c95915b93f659c4759f received from {username='slave'} at 127.13.251.131:45933
I20240501 20:10:11.929610 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-e1be0c0baa5e4c95915b93f659c4759f on tablet e1be0c0baa5e4c95915b93f659c4759f with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:10:11.930486 28813 ts_tablet_manager.cc:902] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:11.948518 28813 tablet_copy_client.cc:308] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:10:11.949184 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet 182bd3935cce49868a451685097e9f0c from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:41549)
I20240501 20:10:11.949329 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet 182bd3935cce49868a451685097e9f0c from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:41549: session id = 61380023001a4ee9ada9ec2c6c1b7159-182bd3935cce49868a451685097e9f0c
I20240501 20:10:11.952692 23997 tablet_copy_source_session.cc:205] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:11.954056 28813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182bd3935cce49868a451685097e9f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.958691 23843 consensus_queue.cc:237] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:11.961627 24110 raft_consensus.cc:1261] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:11.962190 23779 consensus_peers.cc:485] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2382513ca70640319674c35864e22d9c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.963186 28389 consensus_queue.cc:1035] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.964098 23975 raft_consensus.cc:1261] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.964601 28389 consensus_queue.cc:1035] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.980437 28827 ts_tablet_manager.cc:902] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:11.990200 28813 tablet_copy_client.cc:783] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:11.990447 28813 tablet_copy_client.cc:647] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:11.992153 28813 tablet_copy_client.cc:523] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:11.995620 28813 tablet_bootstrap.cc:492] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:12.022194 28813 tablet_bootstrap.cc:492] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.022656 28813 tablet_bootstrap.cc:492] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:10:12.022917 28813 ts_tablet_manager.cc:1367] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.009s
I20240501 20:10:12.024004 28813 raft_consensus.cc:348] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:12.024353 28813 raft_consensus.cc:729] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.024654 28813 consensus_queue.cc:260] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:12.025873 28813 ts_tablet_manager.cc:1398] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:12.027344 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-182bd3935cce49868a451685097e9f0c received from {username='slave'} at 127.13.251.132:41549
I20240501 20:10:12.027453 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-182bd3935cce49868a451685097e9f0c on tablet 182bd3935cce49868a451685097e9f0c with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:10:12.029577 28827 tablet_copy_client.cc:308] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.133:37155
I20240501 20:10:12.032502 28389 raft_consensus.cc:2932] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:12.033061 23734 tablet_copy_service.cc:140] P 51bf640e895b4e5eb91bdf7509ea7e46: Received BeginTabletCopySession request for tablet eb612facbc174e0b957294af03ef2864 from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:51369)
I20240501 20:10:12.033217 23734 tablet_copy_service.cc:161] P 51bf640e895b4e5eb91bdf7509ea7e46: Beginning new tablet copy session on tablet eb612facbc174e0b957294af03ef2864 from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:51369: session id = feac73db0a3b4aafa374ca5dfd322b08-eb612facbc174e0b957294af03ef2864
I20240501 20:10:12.033494 23975 raft_consensus.cc:2932] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:12.036130 23734 tablet_copy_source_session.cc:205] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:12.038522 24110 raft_consensus.cc:2932] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:12.039485 20820 catalog_manager.cc:5484] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:12.042748 28827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb612facbc174e0b957294af03ef2864. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.059345 28827 tablet_copy_client.cc:783] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:12.059621 28827 tablet_copy_client.cc:647] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:12.063040 28827 tablet_copy_client.cc:523] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:12.066648 28827 tablet_bootstrap.cc:492] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:12.078594 28827 tablet_bootstrap.cc:492] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.079018 28827 tablet_bootstrap.cc:492] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:10:12.079288 28827 ts_tablet_manager.cc:1367] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.013s
I20240501 20:10:12.079946 28827 raft_consensus.cc:348] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:12.088275 28827 raft_consensus.cc:729] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.088819 28827 consensus_queue.cc:260] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:12.092870 28827 ts_tablet_manager.cc:1398] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240501 20:10:12.093828 23734 tablet_copy_service.cc:342] P 51bf640e895b4e5eb91bdf7509ea7e46: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-eb612facbc174e0b957294af03ef2864 received from {username='slave'} at 127.13.251.130:51369
I20240501 20:10:12.093987 23734 tablet_copy_service.cc:433] P 51bf640e895b4e5eb91bdf7509ea7e46: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-eb612facbc174e0b957294af03ef2864 on tablet eb612facbc174e0b957294af03ef2864 with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:10:12.133189 23712 consensus_queue.cc:237] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:12.135581 24110 raft_consensus.cc:1203] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:12.135929 23843 raft_consensus.cc:1261] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:12.136569 23975 raft_consensus.cc:1261] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:12.137068 23644 consensus_peers.cc:485] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25dee912ab5f4613b66d056a8731f188. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:12.137387 28096 consensus_queue.cc:1035] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:12.137728 28852 consensus_queue.cc:1035] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:12.142280 28096 raft_consensus.cc:2932] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:12.142782 23843 raft_consensus.cc:2932] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:12.143425 28792 ts_tablet_manager.cc:902] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:12.147238 23975 raft_consensus.cc:2932] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:12.151739 20818 catalog_manager.cc:5484] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:12.175851 28792 tablet_copy_client.cc:308] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:10:12.176586 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet d37fde63d6414745b0f81c0a0abe4f46 from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:45053)
I20240501 20:10:12.176777 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet d37fde63d6414745b0f81c0a0abe4f46 from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:45053: session id = cda19d6a81c0473db302535ce2ada7f0-d37fde63d6414745b0f81c0a0abe4f46
I20240501 20:10:12.179361 23866 tablet_copy_source_session.cc:205] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:12.180886 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37fde63d6414745b0f81c0a0abe4f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.185555 28792 tablet_copy_client.cc:783] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:12.185866 28792 tablet_copy_client.cc:647] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:12.188413 28792 tablet_copy_client.cc:523] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:12.191681 28792 tablet_bootstrap.cc:492] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:12.197569 28792 tablet_bootstrap.cc:492] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.197894 28792 tablet_bootstrap.cc:492] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:10:12.198125 28792 ts_tablet_manager.cc:1367] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20240501 20:10:12.198872 28792 raft_consensus.cc:348] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:12.199115 28792 raft_consensus.cc:729] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.199329 28792 consensus_queue.cc:260] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:12.199839 28792 ts_tablet_manager.cc:1398] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:12.201428 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-d37fde63d6414745b0f81c0a0abe4f46 received from {username='slave'} at 127.13.251.129:45053
I20240501 20:10:12.201578 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-d37fde63d6414745b0f81c0a0abe4f46 on tablet d37fde63d6414745b0f81c0a0abe4f46 with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:12.203099 28827 ts_tablet_manager.cc:902] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:12.207423 24110 consensus_queue.cc:237] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:12.219238 23712 raft_consensus.cc:1261] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:12.220075 28827 tablet_copy_client.cc:308] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
W20240501 20:10:12.220758 24044 consensus_peers.cc:485] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a163a3edf82e45f39b4f2a26f0466e9b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:12.220886 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet c58dd83ebd0949dea6f4567067c8dfc9 from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:58461)
I20240501 20:10:12.221065 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet c58dd83ebd0949dea6f4567067c8dfc9 from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:58461: session id = feac73db0a3b4aafa374ca5dfd322b08-c58dd83ebd0949dea6f4567067c8dfc9
I20240501 20:10:12.224720 23975 raft_consensus.cc:1261] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:12.224767 23866 tablet_copy_source_session.cc:205] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:12.221410 28182 consensus_queue.cc:1035] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240501 20:10:12.234171 28303 consensus_queue.cc:1035] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:12.258330 28883 ts_tablet_manager.cc:902] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:12.259330 28827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58dd83ebd0949dea6f4567067c8dfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.273540 28303 raft_consensus.cc:2932] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:12.274549 28883 tablet_copy_client.cc:308] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:10:12.275168 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 72726d6b786e4c72ace7359101cec949 from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:58619)
I20240501 20:10:12.275298 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 72726d6b786e4c72ace7359101cec949 from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:58619: session id = 8559229e044b4e1ea661b881b3f9af50-72726d6b786e4c72ace7359101cec949
I20240501 20:10:12.275717 28827 tablet_copy_client.cc:783] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:12.275940 28827 tablet_copy_client.cc:647] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:12.277343 28827 tablet_copy_client.cc:523] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:12.277581 21378 tablet_copy_source_session.cc:205] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:12.278576 23712 raft_consensus.cc:2932] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:12.279773 23977 raft_consensus.cc:2932] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:12.287288 28883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72726d6b786e4c72ace7359101cec949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.287706 20819 catalog_manager.cc:5484] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:12.295634 28883 tablet_copy_client.cc:783] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:12.296041 28883 tablet_copy_client.cc:647] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:12.296321 28827 tablet_bootstrap.cc:492] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:12.302763 28883 tablet_copy_client.cc:523] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:12.307575 28883 tablet_bootstrap.cc:492] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:12.310530 28827 tablet_bootstrap.cc:492] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.311641 28827 tablet_bootstrap.cc:492] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:10:12.311904 28827 ts_tablet_manager.cc:1367] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.011s
I20240501 20:10:12.313179 28827 raft_consensus.cc:348] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:12.313387 28827 raft_consensus.cc:729] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.313616 28827 consensus_queue.cc:260] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:12.314095 28827 ts_tablet_manager.cc:1398] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:12.314867 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-c58dd83ebd0949dea6f4567067c8dfc9 received from {username='slave'} at 127.13.251.130:58461
I20240501 20:10:12.315017 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-c58dd83ebd0949dea6f4567067c8dfc9 on tablet c58dd83ebd0949dea6f4567067c8dfc9 with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:10:12.351788 28883 tablet_bootstrap.cc:492] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.352275 28883 tablet_bootstrap.cc:492] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:10:12.352548 28883 ts_tablet_manager.cc:1367] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.045s user 0.035s sys 0.007s
I20240501 20:10:12.353189 28883 raft_consensus.cc:348] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:12.353379 28883 raft_consensus.cc:729] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.353591 28883 consensus_queue.cc:260] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:12.354063 28883 ts_tablet_manager.cc:1398] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:10:12.354763 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-72726d6b786e4c72ace7359101cec949 received from {username='slave'} at 127.13.251.136:58619
I20240501 20:10:12.354861 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-72726d6b786e4c72ace7359101cec949 on tablet 72726d6b786e4c72ace7359101cec949 with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:10:12.360033 28813 ts_tablet_manager.cc:902] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:10:12.360247 20967 raft_consensus.cc:1203] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:12.390209 28813 tablet_copy_client.cc:308] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:10:12.390980 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet 6ffdb911cf1547329552d2e6af92c6df from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:47489)
I20240501 20:10:12.391141 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet 6ffdb911cf1547329552d2e6af92c6df from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:47489: session id = 61380023001a4ee9ada9ec2c6c1b7159-6ffdb911cf1547329552d2e6af92c6df
I20240501 20:10:12.393955 21118 tablet_copy_source_session.cc:205] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:12.395607 28813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffdb911cf1547329552d2e6af92c6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.400564 28813 tablet_copy_client.cc:783] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:12.400843 28813 tablet_copy_client.cc:647] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:12.402643 28813 tablet_copy_client.cc:523] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:12.405386 28813 tablet_bootstrap.cc:492] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:12.410784 28813 tablet_bootstrap.cc:492] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.411175 28813 tablet_bootstrap.cc:492] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:10:12.411407 28813 ts_tablet_manager.cc:1367] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:10:12.412164 28813 raft_consensus.cc:348] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:12.412395 28813 raft_consensus.cc:729] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.412616 28813 consensus_queue.cc:260] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:12.413100 28813 ts_tablet_manager.cc:1398] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:12.414196 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-6ffdb911cf1547329552d2e6af92c6df received from {username='slave'} at 127.13.251.132:47489
I20240501 20:10:12.414340 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-6ffdb911cf1547329552d2e6af92c6df on tablet 6ffdb911cf1547329552d2e6af92c6df with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:10:12.474503 28792 ts_tablet_manager.cc:902] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:12.505178 21228 raft_consensus.cc:1203] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:12.515920 21098 raft_consensus.cc:1203] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:12.533622 21357 raft_consensus.cc:1203] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:12.578763 20967 raft_consensus.cc:1203] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:12.597438 28792 tablet_copy_client.cc:308] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:10:12.598183 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet 2382513ca70640319674c35864e22d9c from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:45053)
I20240501 20:10:12.598366 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet 2382513ca70640319674c35864e22d9c from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:45053: session id = cda19d6a81c0473db302535ce2ada7f0-2382513ca70640319674c35864e22d9c
I20240501 20:10:12.601230 23866 tablet_copy_source_session.cc:205] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:12.602480 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2382513ca70640319674c35864e22d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.611809 28792 tablet_copy_client.cc:783] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:12.612032 28792 tablet_copy_client.cc:647] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:12.613749 28792 tablet_copy_client.cc:523] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:12.618022 28792 tablet_bootstrap.cc:492] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:12.623675 28792 tablet_bootstrap.cc:492] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.623960 28792 tablet_bootstrap.cc:492] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:10:12.624161 28792 ts_tablet_manager.cc:1367] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.001s
I20240501 20:10:12.624750 28792 raft_consensus.cc:348] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:12.624954 28792 raft_consensus.cc:729] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.625164 28792 consensus_queue.cc:260] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:12.625823 28792 ts_tablet_manager.cc:1398] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:12.626487 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-2382513ca70640319674c35864e22d9c received from {username='slave'} at 127.13.251.129:45053
I20240501 20:10:12.626608 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-2382513ca70640319674c35864e22d9c on tablet 2382513ca70640319674c35864e22d9c with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:12.629978 28827 ts_tablet_manager.cc:902] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:12.656235 28827 tablet_copy_client.cc:308] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.133:37155
I20240501 20:10:12.656958 23734 tablet_copy_service.cc:140] P 51bf640e895b4e5eb91bdf7509ea7e46: Received BeginTabletCopySession request for tablet 25dee912ab5f4613b66d056a8731f188 from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:51369)
I20240501 20:10:12.657099 23734 tablet_copy_service.cc:161] P 51bf640e895b4e5eb91bdf7509ea7e46: Beginning new tablet copy session on tablet 25dee912ab5f4613b66d056a8731f188 from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:51369: session id = feac73db0a3b4aafa374ca5dfd322b08-25dee912ab5f4613b66d056a8731f188
I20240501 20:10:12.667079 23734 tablet_copy_source_session.cc:205] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:12.671880 24110 raft_consensus.cc:1203] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:12.676079 28827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25dee912ab5f4613b66d056a8731f188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.681866 28827 tablet_copy_client.cc:783] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:12.682096 28827 tablet_copy_client.cc:647] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:12.684654 28827 tablet_copy_client.cc:523] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:12.688052 28827 tablet_bootstrap.cc:492] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:12.701934 28827 tablet_bootstrap.cc:492] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.702307 28827 tablet_bootstrap.cc:492] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:10:12.702558 28827 ts_tablet_manager.cc:1367] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240501 20:10:12.703224 28827 raft_consensus.cc:348] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:12.703431 28827 raft_consensus.cc:729] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.709507 28827 consensus_queue.cc:260] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:12.710079 28827 ts_tablet_manager.cc:1398] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240501 20:10:12.711212 23734 tablet_copy_service.cc:342] P 51bf640e895b4e5eb91bdf7509ea7e46: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-25dee912ab5f4613b66d056a8731f188 received from {username='slave'} at 127.13.251.130:51369
I20240501 20:10:12.711341 23734 tablet_copy_service.cc:433] P 51bf640e895b4e5eb91bdf7509ea7e46: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-25dee912ab5f4613b66d056a8731f188 on tablet 25dee912ab5f4613b66d056a8731f188 with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:10:12.716953 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:12.739748 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:12.767270 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:12.786792 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:12.793648 21098 raft_consensus.cc:1203] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:12.822630 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:12.830118 28813 ts_tablet_manager.cc:902] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:12.830478 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:12.848855 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:12.862874 20967 raft_consensus.cc:1203] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:12.882880 21357 raft_consensus.cc:1203] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:12.897142 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:12.905987 28813 tablet_copy_client.cc:308] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.136:44267
I20240501 20:10:12.911209 24130 tablet_copy_service.cc:140] P 8559229e044b4e1ea661b881b3f9af50: Received BeginTabletCopySession request for tablet a163a3edf82e45f39b4f2a26f0466e9b from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:52271)
I20240501 20:10:12.911430 24130 tablet_copy_service.cc:161] P 8559229e044b4e1ea661b881b3f9af50: Beginning new tablet copy session on tablet a163a3edf82e45f39b4f2a26f0466e9b from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:52271: session id = 61380023001a4ee9ada9ec2c6c1b7159-a163a3edf82e45f39b4f2a26f0466e9b
I20240501 20:10:12.923195 24130 tablet_copy_source_session.cc:205] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:12.930194 28813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a163a3edf82e45f39b4f2a26f0466e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.943163 28813 tablet_copy_client.cc:783] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:12.943437 28813 tablet_copy_client.cc:647] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:12.949420 28813 tablet_copy_client.cc:523] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:12.962460 28813 tablet_bootstrap.cc:492] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:13.000139 28813 tablet_bootstrap.cc:492] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:13.000550 28813 tablet_bootstrap.cc:492] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:10:13.000778 28813 ts_tablet_manager.cc:1367] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.000s
I20240501 20:10:13.001511 28813 raft_consensus.cc:348] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:13.001713 28813 raft_consensus.cc:729] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.001907 28813 consensus_queue.cc:260] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:13.002416 28813 ts_tablet_manager.cc:1398] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:13.004765 24130 tablet_copy_service.cc:342] P 8559229e044b4e1ea661b881b3f9af50: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-a163a3edf82e45f39b4f2a26f0466e9b received from {username='slave'} at 127.13.251.132:52271
I20240501 20:10:13.004925 24130 tablet_copy_service.cc:433] P 8559229e044b4e1ea661b881b3f9af50: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-a163a3edf82e45f39b4f2a26f0466e9b on tablet a163a3edf82e45f39b4f2a26f0466e9b with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:10:13.147367 21098 raft_consensus.cc:1203] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:13.293417 21357 raft_consensus.cc:1203] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:13.314584 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:13.340718 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:13.383783 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:13.383883 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:13.420416 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:13.425009 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:13.440029 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:13.489938 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:13.667477 23842 consensus_queue.cc:237] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:13.669629 24110 raft_consensus.cc:1261] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:13.669924 20967 raft_consensus.cc:1261] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:13.670248 28389 consensus_queue.cc:1035] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:13.670568 28649 consensus_queue.cc:1035] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:13.673854 28389 raft_consensus.cc:2932] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:13.675295 20967 raft_consensus.cc:2932] T 2382513ca70640319674c35864e22d9c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:13.675642 24110 raft_consensus.cc:2932] T 2382513ca70640319674c35864e22d9c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:13.681607 20819 catalog_manager.cc:5484] T 2382513ca70640319674c35864e22d9c P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:13.691151 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 2382513ca70640319674c35864e22d9c with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:13.691840 29073 tablet_replica.cc:330] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:13.692473 29073 raft_consensus.cc:2229] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:13.692719 29073 raft_consensus.cc:2258] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:13.693432 29073 ts_tablet_manager.cc:1875] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:13.700871 29073 ts_tablet_manager.cc:1888] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:13.701010 29073 log.cc:1196] T 2382513ca70640319674c35864e22d9c P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/2382513ca70640319674c35864e22d9c
I20240501 20:10:13.701738 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 2382513ca70640319674c35864e22d9c (table rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]) successfully deleted
I20240501 20:10:14.081769 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:14.083298 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:14.092320 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:14.111886 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:14.147274 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:14.149195 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:14.199033 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:14.211280 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:14.371519 21358 consensus_queue.cc:237] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:14.374217 24109 raft_consensus.cc:1261] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:14.374670 28098 consensus_queue.cc:1035] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:14.376428 21095 raft_consensus.cc:1261] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:14.377107 28789 consensus_queue.cc:1035] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:14.379627 28789 raft_consensus.cc:2932] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:14.382988 24109 raft_consensus.cc:2932] T 72726d6b786e4c72ace7359101cec949 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:14.387903 20818 catalog_manager.cc:5484] T 72726d6b786e4c72ace7359101cec949 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:14.395278 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet 72726d6b786e4c72ace7359101cec949 with delete_type TABLET_DATA_TOMBSTONED (TS 4af096f5a6a9440a94f6682b399efe20 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45322
I20240501 20:10:14.398947 29165 tablet_replica.cc:330] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:10:14.399916 29165 raft_consensus.cc:2229] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:14.400501 21095 raft_consensus.cc:2932] T 72726d6b786e4c72ace7359101cec949 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:14.401068 29165 raft_consensus.cc:2258] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:14.401816 29165 ts_tablet_manager.cc:1875] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:14.408360 29165 ts_tablet_manager.cc:1888] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:14.408543 29165 log.cc:1196] T 72726d6b786e4c72ace7359101cec949 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/72726d6b786e4c72ace7359101cec949
I20240501 20:10:14.409271 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 72726d6b786e4c72ace7359101cec949 (table rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]) successfully deleted
I20240501 20:10:14.754571 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:14.794358 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:14.795434 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:14.828800 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:14.858307 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:14.864384 21077 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:14.871976 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240501 20:10:14.876214 21034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 65513.7us
I20240501 20:10:14.892602 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:15.080132 23843 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9"
dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57672
I20240501 20:10:15.080363 23843 raft_consensus.cc:595] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:15.323035 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:15.378737 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:15.385921 23825 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:15.386915 29130 raft_consensus.cc:979] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: : Instructing follower feac73db0a3b4aafa374ca5dfd322b08 to start an election
I20240501 20:10:15.387042 29130 raft_consensus.cc:1067] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Signalling peer feac73db0a3b4aafa374ca5dfd322b08 to start an election
I20240501 20:10:15.393574 21098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9"
dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
from {username='slave'} at 127.13.251.134:55907
I20240501 20:10:15.393846 21098 raft_consensus.cc:482] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:15.393947 21098 raft_consensus.cc:3037] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.421072 21098 raft_consensus.cc:504] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:15.422578 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:15.426610 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:15.444038 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:10:15.445087 21228 raft_consensus.cc:3037] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.452610 23712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:10:15.452921 23712 raft_consensus.cc:3037] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.457572 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:15.463074 23712 raft_consensus.cc:2454] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:10:15.487243 21098 leader_election.cc:288] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:15.489010 23842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58dd83ebd0949dea6f4567067c8dfc9" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:10:15.489228 23842 raft_consensus.cc:3032] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:15.489297 23842 raft_consensus.cc:729] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:10:15.489526 23842 consensus_queue.cc:260] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:15.489943 23842 raft_consensus.cc:3037] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.493105 23842 raft_consensus.cc:2454] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:10:15.493101 21228 raft_consensus.cc:2454] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:10:15.493523 21034 leader_election.cc:302] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:10:15.494107 29221 raft_consensus.cc:2781] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:15.494239 29221 raft_consensus.cc:686] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:10:15.494514 29221 consensus_queue.cc:237] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:15.502893 20824 catalog_manager.cc:5484] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 1 to 2, leader changed from c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 2 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:15.503875 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:15.544520 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:15.936995 21228 raft_consensus.cc:1261] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:15.938333 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:15.938352 29221 consensus_queue.cc:1035] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:16.001761 23842 raft_consensus.cc:1261] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:16.002427 29221 consensus_queue.cc:1035] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:16.026129 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:16.032539 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:16.052554 23709 raft_consensus.cc:1261] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:16.053673 29221 consensus_queue.cc:1035] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:16.209371 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:16.262130 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:16.276497 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:16.311306 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:16.339974 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:16.448948 21098 consensus_queue.cc:237] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:16.451208 21228 raft_consensus.cc:1261] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:16.451249 23843 raft_consensus.cc:1261] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:16.451812 28710 consensus_queue.cc:1035] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:16.452080 29356 consensus_queue.cc:1035] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:16.454847 29221 raft_consensus.cc:2932] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:16.455461 23843 raft_consensus.cc:2932] T e1be0c0baa5e4c95915b93f659c4759f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:16.462284 20825 catalog_manager.cc:5484] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:16.466601 21228 raft_consensus.cc:2932] T e1be0c0baa5e4c95915b93f659c4759f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
W20240501 20:10:16.507344 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 48559.4us
I20240501 20:10:16.556171 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet e1be0c0baa5e4c95915b93f659c4759f with delete_type TABLET_DATA_TOMBSTONED (TS 8559229e044b4e1ea661b881b3f9af50 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:16.567976 29398 tablet_replica.cc:330] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:16.568325 29398 raft_consensus.cc:2229] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:16.568624 29398 raft_consensus.cc:2258] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:16.569388 29398 ts_tablet_manager.cc:1875] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:16.584141 29398 ts_tablet_manager.cc:1888] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:16.584429 29398 log.cc:1196] T e1be0c0baa5e4c95915b93f659c4759f P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/e1be0c0baa5e4c95915b93f659c4759f
I20240501 20:10:16.585887 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet e1be0c0baa5e4c95915b93f659c4759f (table rebalancer_extra_table_26 [id=26f0e575ea3f478e8c65c2d3553d8896]) successfully deleted
I20240501 20:10:16.833002 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:16.883607 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:16.906056 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:16.929625 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:17.015825 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:17.054512 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:17.111783 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:17.144989 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:17.240142 23975 consensus_queue.cc:237] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:17.244789 21356 raft_consensus.cc:1261] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:17.244951 20963 raft_consensus.cc:1261] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:17.245569 29441 consensus_queue.cc:1035] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:17.247242 29355 consensus_queue.cc:1035] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:17.252264 29440 raft_consensus.cc:2932] T 182bd3935cce49868a451685097e9f0c P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:17.253011 20963 raft_consensus.cc:2932] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:17.261447 20825 catalog_manager.cc:5484] T 182bd3935cce49868a451685097e9f0c P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:17.269274 21358 raft_consensus.cc:2932] T 182bd3935cce49868a451685097e9f0c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:17.279560 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 182bd3935cce49868a451685097e9f0c with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:17.284373 29484 tablet_replica.cc:330] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:17.284740 29484 raft_consensus.cc:2229] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:17.284998 29484 raft_consensus.cc:2258] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:17.285847 29484 ts_tablet_manager.cc:1875] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:17.292943 29484 ts_tablet_manager.cc:1888] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:17.293124 29484 log.cc:1196] T 182bd3935cce49868a451685097e9f0c P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/182bd3935cce49868a451685097e9f0c
I20240501 20:10:17.294252 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 182bd3935cce49868a451685097e9f0c (table rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]) successfully deleted
I20240501 20:10:17.579588 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:17.586578 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:17.632208 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:17.689286 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:17.740371 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:17.806202 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:17.821722 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:17.822314 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:18.039428 21095 consensus_queue.cc:237] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:18.049242 21358 raft_consensus.cc:1261] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:18.049891 29221 consensus_queue.cc:1035] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:18.051122 23975 raft_consensus.cc:1261] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:18.051687 29221 consensus_queue.cc:1035] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:18.064373 29221 raft_consensus.cc:2932] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:18.068492 23975 raft_consensus.cc:2932] T 6ffdb911cf1547329552d2e6af92c6df P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:18.074159 20820 catalog_manager.cc:5484] T 6ffdb911cf1547329552d2e6af92c6df P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:18.077896 21358 raft_consensus.cc:2932] T 6ffdb911cf1547329552d2e6af92c6df P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:18.091066 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ffdb911cf1547329552d2e6af92c6df with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:18.100454 29563 tablet_replica.cc:330] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:18.100687 29563 raft_consensus.cc:2229] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:18.100889 29563 raft_consensus.cc:2258] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:18.101773 29563 ts_tablet_manager.cc:1875] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:18.108237 29563 ts_tablet_manager.cc:1888] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:18.108546 29563 log.cc:1196] T 6ffdb911cf1547329552d2e6af92c6df P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/6ffdb911cf1547329552d2e6af92c6df
I20240501 20:10:18.109299 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 6ffdb911cf1547329552d2e6af92c6df (table rebalancer_extra_table_38 [id=249e43918dea42f8b8357e313d7799fd]) successfully deleted
I20240501 20:10:18.352712 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:18.353925 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:18.393102 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:18.394122 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:18.395797 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:18.398607 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:18.408457 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:18.428192 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:10:18.534535 23911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 71317.5us
I20240501 20:10:18.660488 23709 consensus_queue.cc:237] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:18.662422 21095 raft_consensus.cc:1261] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:18.662460 24108 raft_consensus.cc:1261] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:18.663012 29385 consensus_queue.cc:1035] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:18.663316 29472 consensus_queue.cc:1035] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:18.666451 29472 raft_consensus.cc:2932] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:18.667518 21095 raft_consensus.cc:2932] T eb612facbc174e0b957294af03ef2864 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:18.669947 24108 raft_consensus.cc:2932] T eb612facbc174e0b957294af03ef2864 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:18.673209 20820 catalog_manager.cc:5484] T eb612facbc174e0b957294af03ef2864 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:18.707459 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet eb612facbc174e0b957294af03ef2864 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:10:18.719897 29657 tablet_replica.cc:330] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:10:18.720067 29657 raft_consensus.cc:2229] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:18.720202 29657 raft_consensus.cc:2258] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:18.720671 29657 ts_tablet_manager.cc:1875] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:18.724776 29657 ts_tablet_manager.cc:1888] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:18.724864 29657 log.cc:1196] T eb612facbc174e0b957294af03ef2864 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/eb612facbc174e0b957294af03ef2864
I20240501 20:10:18.725468 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet eb612facbc174e0b957294af03ef2864 (table rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]) successfully deleted
I20240501 20:10:18.925673 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:18.980954 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:19.005496 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:19.020162 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:19.078485 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:19.079164 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:19.116572 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:19.140497 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:19.282827 23846 consensus_queue.cc:237] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:19.284704 24108 raft_consensus.cc:1261] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:19.284749 20963 raft_consensus.cc:1261] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:19.285187 29449 consensus_queue.cc:1035] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:19.285493 29705 consensus_queue.cc:1035] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:19.288401 29449 raft_consensus.cc:2932] T d37fde63d6414745b0f81c0a0abe4f46 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:19.288914 20963 raft_consensus.cc:2932] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:19.293892 24108 raft_consensus.cc:2932] T d37fde63d6414745b0f81c0a0abe4f46 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:19.295444 20817 catalog_manager.cc:5484] T d37fde63d6414745b0f81c0a0abe4f46 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:19.303114 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet d37fde63d6414745b0f81c0a0abe4f46 with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:19.319922 29728 tablet_replica.cc:330] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:19.320617 29728 raft_consensus.cc:2229] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:19.320866 29728 raft_consensus.cc:2258] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:19.321594 29728 ts_tablet_manager.cc:1875] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:19.330821 29728 ts_tablet_manager.cc:1888] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:19.331162 29728 log.cc:1196] T d37fde63d6414745b0f81c0a0abe4f46 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/d37fde63d6414745b0f81c0a0abe4f46
I20240501 20:10:19.332265 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet d37fde63d6414745b0f81c0a0abe4f46 (table rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]) successfully deleted
I20240501 20:10:19.661824 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:19.676505 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:19.697963 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:19.719380 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:19.737180 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:19.746614 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:19.778793 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:19.804272 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:19.929632 23977 consensus_queue.cc:237] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:19.932322 21098 raft_consensus.cc:1261] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:19.932384 20967 raft_consensus.cc:1261] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:19.932933 29621 consensus_queue.cc:1035] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:19.933202 29355 consensus_queue.cc:1035] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:19.936645 29621 raft_consensus.cc:2932] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:19.940479 21098 raft_consensus.cc:2932] T d8ef72f9e1d04bb498f540a74a850ed8 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:19.947292 20821 catalog_manager.cc:5484] T d8ef72f9e1d04bb498f540a74a850ed8 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:19.957932 20967 raft_consensus.cc:2932] T d8ef72f9e1d04bb498f540a74a850ed8 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:19.959846 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ef72f9e1d04bb498f540a74a850ed8 with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:19.960785 29823 tablet_replica.cc:330] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:19.960997 29823 raft_consensus.cc:2229] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:19.961282 29823 raft_consensus.cc:2258] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:19.962042 29823 ts_tablet_manager.cc:1875] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:19.971855 29823 ts_tablet_manager.cc:1888] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:19.972079 29823 log.cc:1196] T d8ef72f9e1d04bb498f540a74a850ed8 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/d8ef72f9e1d04bb498f540a74a850ed8
I20240501 20:10:19.972903 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet d8ef72f9e1d04bb498f540a74a850ed8 (table rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]) successfully deleted
I20240501 20:10:20.270963 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:20.291288 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:20.311525 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:20.330250 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:20.332664 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:20.354590 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:20.398177 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:20.449613 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:20.602891 23846 consensus_queue.cc:237] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:20.605254 20967 raft_consensus.cc:1261] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:20.605485 24110 raft_consensus.cc:1261] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:20.605801 29851 consensus_queue.cc:1035] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:20.606091 28649 consensus_queue.cc:1035] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:20.611850 29907 raft_consensus.cc:2932] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:20.612141 20967 raft_consensus.cc:2932] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:20.621551 24110 raft_consensus.cc:2932] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:20.621737 20822 catalog_manager.cc:5484] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:20.638626 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9 with delete_type TABLET_DATA_TOMBSTONED (TS 4af096f5a6a9440a94f6682b399efe20 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45322
I20240501 20:10:20.645352 29914 tablet_replica.cc:330] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:10:20.648794 29914 raft_consensus.cc:2229] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:20.649310 29914 raft_consensus.cc:2258] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:20.649941 29914 ts_tablet_manager.cc:1875] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:20.655975 29914 ts_tablet_manager.cc:1888] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:20.656108 29914 log.cc:1196] T ad72fa9eb71c4a2f9eaee8f48f7bbca9 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/ad72fa9eb71c4a2f9eaee8f48f7bbca9
I20240501 20:10:20.656787 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet ad72fa9eb71c4a2f9eaee8f48f7bbca9 (table rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]) successfully deleted
I20240501 20:10:20.862282 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:20.929828 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:20.939208 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:20.940918 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:20.986137 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:20.996894 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:21.032913 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:21.036522 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:21.197703 20967 consensus_queue.cc:237] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:21.200207 24110 raft_consensus.cc:1261] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:21.200240 21095 raft_consensus.cc:1261] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:21.200682 29219 consensus_queue.cc:1035] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:21.201155 29219 consensus_queue.cc:1035] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:21.203778 29219 raft_consensus.cc:2932] T 75fa04c8726c4f2e932740cc6fbdcdd0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:21.204320 21097 raft_consensus.cc:2932] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:21.210398 20822 catalog_manager.cc:5484] T 75fa04c8726c4f2e932740cc6fbdcdd0 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:21.216044 24110 raft_consensus.cc:2932] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:21.227448 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet 75fa04c8726c4f2e932740cc6fbdcdd0 with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42940
I20240501 20:10:21.237917 29999 tablet_replica.cc:330] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:10:21.238171 29999 raft_consensus.cc:2229] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:21.238663 29999 raft_consensus.cc:2258] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:21.239408 29999 ts_tablet_manager.cc:1875] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:21.246199 29999 ts_tablet_manager.cc:1888] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:21.246419 29999 log.cc:1196] T 75fa04c8726c4f2e932740cc6fbdcdd0 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/75fa04c8726c4f2e932740cc6fbdcdd0
I20240501 20:10:21.247160 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 75fa04c8726c4f2e932740cc6fbdcdd0 (table rebalance_test_table_11 [id=e0b22a22ea984e318d2e5799a0d66e3f]) successfully deleted
I20240501 20:10:21.486521 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:21.495317 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:21.502995 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:21.530058 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:21.546195 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:21.584316 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:21.615123 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:21.645843 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:21.790642 24109 consensus_queue.cc:237] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:21.793248 21356 raft_consensus.cc:1261] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.793407 23977 raft_consensus.cc:1261] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.793752 29030 consensus_queue.cc:1035] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.794123 29885 consensus_queue.cc:1035] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.796602 29885 raft_consensus.cc:2932] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:21.797230 21356 raft_consensus.cc:2932] T a163a3edf82e45f39b4f2a26f0466e9b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:21.798106 23977 raft_consensus.cc:2932] T a163a3edf82e45f39b4f2a26f0466e9b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:21.805686 20824 catalog_manager.cc:5484] T a163a3edf82e45f39b4f2a26f0466e9b P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:21.815299 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet a163a3edf82e45f39b4f2a26f0466e9b with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:21.817504 30090 tablet_replica.cc:330] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:21.817799 30090 raft_consensus.cc:2229] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:21.818068 30090 raft_consensus.cc:2258] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:21.819085 30090 ts_tablet_manager.cc:1875] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:21.824975 30090 ts_tablet_manager.cc:1888] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:21.825089 30090 log.cc:1196] T a163a3edf82e45f39b4f2a26f0466e9b P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/a163a3edf82e45f39b4f2a26f0466e9b
I20240501 20:10:21.825789 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet a163a3edf82e45f39b4f2a26f0466e9b (table rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]) successfully deleted
I20240501 20:10:21.885303 23711 consensus_queue.cc:237] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:21.887277 21225 raft_consensus.cc:1261] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:21.887509 23644 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fda032f2be744fe95170b3c3287d6ae. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:21.887660 23977 raft_consensus.cc:1261] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:21.887748 29886 consensus_queue.cc:1035] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:21.888103 29642 consensus_queue.cc:1035] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:21.890578 29642 raft_consensus.cc:2932] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:21.891249 23977 raft_consensus.cc:2932] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:21.894891 21225 raft_consensus.cc:2932] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:21.897694 20823 catalog_manager.cc:5484] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:21.993961 20963 consensus_queue.cc:237] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:21.997586 21356 raft_consensus.cc:1261] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:21.998304 21225 raft_consensus.cc:1261] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:21.999855 30111 consensus_queue.cc:1035] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:22.000734 20903 consensus_peers.cc:485] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 -> Peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Couldn't send request to peer c6dcaa8efff642dcb7c2c6e8be3a65f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7eb4c409a2945f0b39bebd37fec4ea7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.001001 30111 consensus_queue.cc:1035] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.015136 30111 raft_consensus.cc:2932] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:22.020093 21225 raft_consensus.cc:2932] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:22.022969 21356 raft_consensus.cc:2932] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:22.023573 20820 catalog_manager.cc:5484] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index -1 to 2, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:22.099107 21225 consensus_queue.cc:237] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:22.101805 23977 raft_consensus.cc:1261] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:22.102030 21163 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 718853c0da3d4a68ab478a6e79cdd438. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.102316 20963 raft_consensus.cc:1261] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.102411 29446 consensus_queue.cc:1035] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.103205 29446 consensus_queue.cc:1035] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.112807 30062 raft_consensus.cc:2932] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:22.113709 20963 raft_consensus.cc:2932] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:22.117564 23977 raft_consensus.cc:2932] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:22.119832 20824 catalog_manager.cc:5484] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index -1 to 2, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:22.165076 23711 consensus_queue.cc:237] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
W20240501 20:10:22.168186 23645 consensus_peers.cc:485] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83384d743f874d77bc8a12d1c9834736. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.168200 23977 raft_consensus.cc:1261] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.168336 20963 raft_consensus.cc:1261] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.168761 29886 consensus_queue.cc:1035] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.169095 30120 consensus_queue.cc:1035] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.174211 30120 raft_consensus.cc:2932] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:22.174783 20963 raft_consensus.cc:2932] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:22.179340 23974 raft_consensus.cc:2932] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:22.180919 20823 catalog_manager.cc:5484] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:22.215312 23846 consensus_queue.cc:237] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:22.217919 23711 raft_consensus.cc:1261] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:22.218407 23779 consensus_peers.cc:485] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c31dffb0482438ebbc6243ad54752e9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.219363 29907 consensus_queue.cc:1035] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.219623 23974 raft_consensus.cc:1261] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.220495 29907 consensus_queue.cc:1035] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.234133 29907 raft_consensus.cc:2932] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:22.237581 23974 raft_consensus.cc:2932] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:22.238924 23711 raft_consensus.cc:2932] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:22.242188 20824 catalog_manager.cc:5484] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:22.289372 21225 consensus_queue.cc:237] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:22.291764 24109 raft_consensus.cc:1261] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.291775 23974 raft_consensus.cc:1261] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:22.292137 21164 consensus_peers.cc:485] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 -> Peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Couldn't send request to peer c6dcaa8efff642dcb7c2c6e8be3a65f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8978a4c14f0a4446a979d3fc05248085. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.292344 30159 consensus_queue.cc:1035] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.292593 29446 consensus_queue.cc:1035] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.295275 29446 raft_consensus.cc:2932] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:22.296006 23974 raft_consensus.cc:2932] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:22.298907 24109 raft_consensus.cc:2932] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:22.302515 20823 catalog_manager.cc:5484] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 2, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:22.306030 30216 ts_tablet_manager.cc:902] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:22.319890 30216 tablet_copy_client.cc:308] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.133:37155
I20240501 20:10:22.320842 23734 tablet_copy_service.cc:140] P 51bf640e895b4e5eb91bdf7509ea7e46: Received BeginTabletCopySession request for tablet 0fda032f2be744fe95170b3c3287d6ae from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:51369)
I20240501 20:10:22.321041 23734 tablet_copy_service.cc:161] P 51bf640e895b4e5eb91bdf7509ea7e46: Beginning new tablet copy session on tablet 0fda032f2be744fe95170b3c3287d6ae from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:51369: session id = feac73db0a3b4aafa374ca5dfd322b08-0fda032f2be744fe95170b3c3287d6ae
I20240501 20:10:22.323489 23734 tablet_copy_source_session.cc:205] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.324906 30216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fda032f2be744fe95170b3c3287d6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.333357 30216 tablet_copy_client.cc:783] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:22.334144 30216 tablet_copy_client.cc:647] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:22.336915 30216 tablet_copy_client.cc:523] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:22.341615 30216 tablet_bootstrap.cc:492] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:22.346873 30216 tablet_bootstrap.cc:492] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:22.347195 30216 tablet_bootstrap.cc:492] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:10:22.347433 30216 ts_tablet_manager.cc:1367] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:10:22.348073 30216 raft_consensus.cc:348] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:22.348325 30216 raft_consensus.cc:729] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.348531 30216 consensus_queue.cc:260] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:22.349102 30216 ts_tablet_manager.cc:1398] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:22.349920 23734 tablet_copy_service.cc:342] P 51bf640e895b4e5eb91bdf7509ea7e46: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-0fda032f2be744fe95170b3c3287d6ae received from {username='slave'} at 127.13.251.130:51369
I20240501 20:10:22.350039 23734 tablet_copy_service.cc:433] P 51bf640e895b4e5eb91bdf7509ea7e46: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-0fda032f2be744fe95170b3c3287d6ae on tablet 0fda032f2be744fe95170b3c3287d6ae with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:10:22.438494 30254 ts_tablet_manager.cc:902] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Initiating tablet copy from peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:22.440287 30254 tablet_copy_client.cc:308] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:41259
I20240501 20:10:22.440949 20987 tablet_copy_service.cc:140] P cda19d6a81c0473db302535ce2ada7f0: Received BeginTabletCopySession request for tablet b7eb4c409a2945f0b39bebd37fec4ea7 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 ({username='slave'} at 127.13.251.134:47553)
I20240501 20:10:22.441090 20987 tablet_copy_service.cc:161] P cda19d6a81c0473db302535ce2ada7f0: Beginning new tablet copy session on tablet b7eb4c409a2945f0b39bebd37fec4ea7 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 at {username='slave'} at 127.13.251.134:47553: session id = c6dcaa8efff642dcb7c2c6e8be3a65f7-b7eb4c409a2945f0b39bebd37fec4ea7
I20240501 20:10:22.443766 20987 tablet_copy_source_session.cc:205] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.445472 30254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7eb4c409a2945f0b39bebd37fec4ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.458992 30254 tablet_copy_client.cc:783] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:22.459399 30254 tablet_copy_client.cc:647] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:22.461411 30254 tablet_copy_client.cc:523] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:22.468009 30254 tablet_bootstrap.cc:492] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:22.490830 30254 tablet_bootstrap.cc:492] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:22.491467 30254 tablet_bootstrap.cc:492] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap complete.
I20240501 20:10:22.491760 30254 ts_tablet_manager.cc:1367] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.004s
I20240501 20:10:22.492462 30254 raft_consensus.cc:348] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:22.492657 30254 raft_consensus.cc:729] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.492931 30254 consensus_queue.cc:260] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:22.493460 30254 ts_tablet_manager.cc:1398] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:22.494275 20987 tablet_copy_service.cc:342] P cda19d6a81c0473db302535ce2ada7f0: Request end of tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-b7eb4c409a2945f0b39bebd37fec4ea7 received from {username='slave'} at 127.13.251.134:47553
I20240501 20:10:22.494382 20987 tablet_copy_service.cc:433] P cda19d6a81c0473db302535ce2ada7f0: ending tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-b7eb4c409a2945f0b39bebd37fec4ea7 on tablet b7eb4c409a2945f0b39bebd37fec4ea7 with peer c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:10:22.638005 30216 ts_tablet_manager.cc:902] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:22.666115 30280 ts_tablet_manager.cc:902] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
W20240501 20:10:22.672622 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 86080.8us
I20240501 20:10:22.674289 30280 tablet_copy_client.cc:308] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.133:37155
I20240501 20:10:22.675695 30279 ts_tablet_manager.cc:902] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:22.682673 30254 ts_tablet_manager.cc:902] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:22.682963 23734 tablet_copy_service.cc:140] P 51bf640e895b4e5eb91bdf7509ea7e46: Received BeginTabletCopySession request for tablet 83384d743f874d77bc8a12d1c9834736 from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:45435)
I20240501 20:10:22.683142 23734 tablet_copy_service.cc:161] P 51bf640e895b4e5eb91bdf7509ea7e46: Beginning new tablet copy session on tablet 83384d743f874d77bc8a12d1c9834736 from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:45435: session id = 61380023001a4ee9ada9ec2c6c1b7159-83384d743f874d77bc8a12d1c9834736
I20240501 20:10:22.685722 23734 tablet_copy_source_session.cc:205] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.688390 30280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83384d743f874d77bc8a12d1c9834736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.694334 30280 tablet_copy_client.cc:783] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:22.694525 30280 tablet_copy_client.cc:647] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:22.697057 30254 tablet_copy_client.cc:308] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:10:22.697288 30280 tablet_copy_client.cc:523] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:22.697794 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet 8978a4c14f0a4446a979d3fc05248085 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 ({username='slave'} at 127.13.251.134:60949)
I20240501 20:10:22.697974 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet 8978a4c14f0a4446a979d3fc05248085 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 at {username='slave'} at 127.13.251.134:60949: session id = c6dcaa8efff642dcb7c2c6e8be3a65f7-8978a4c14f0a4446a979d3fc05248085
I20240501 20:10:22.699259 30216 tablet_copy_client.cc:308] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:10:22.699890 30280 tablet_bootstrap.cc:492] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:22.699903 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet 6c31dffb0482438ebbc6243ad54752e9 from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:58461)
I20240501 20:10:22.700055 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet 6c31dffb0482438ebbc6243ad54752e9 from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:58461: session id = feac73db0a3b4aafa374ca5dfd322b08-6c31dffb0482438ebbc6243ad54752e9
I20240501 20:10:22.701174 21248 tablet_copy_source_session.cc:205] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.702476 30254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8978a4c14f0a4446a979d3fc05248085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.702772 23866 tablet_copy_source_session.cc:205] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.704025 30216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c31dffb0482438ebbc6243ad54752e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.704644 30280 tablet_bootstrap.cc:492] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:22.705621 30280 tablet_bootstrap.cc:492] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:10:22.705838 30280 ts_tablet_manager.cc:1367] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:10:22.706533 30280 raft_consensus.cc:348] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:22.706738 30280 raft_consensus.cc:729] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.708703 30280 consensus_queue.cc:260] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:22.709152 30254 tablet_copy_client.cc:783] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:22.709554 30254 tablet_copy_client.cc:647] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:22.709667 30280 ts_tablet_manager.cc:1398] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:22.710309 23734 tablet_copy_service.cc:342] P 51bf640e895b4e5eb91bdf7509ea7e46: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-83384d743f874d77bc8a12d1c9834736 received from {username='slave'} at 127.13.251.132:45435
I20240501 20:10:22.710438 23734 tablet_copy_service.cc:433] P 51bf640e895b4e5eb91bdf7509ea7e46: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-83384d743f874d77bc8a12d1c9834736 on tablet 83384d743f874d77bc8a12d1c9834736 with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:10:22.711581 30254 tablet_copy_client.cc:523] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:22.717453 30216 tablet_copy_client.cc:783] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:22.715727 30279 tablet_copy_client.cc:308] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:10:22.718070 30216 tablet_copy_client.cc:647] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:22.718477 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet 718853c0da3d4a68ab478a6e79cdd438 from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:42501)
I20240501 20:10:22.718634 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet 718853c0da3d4a68ab478a6e79cdd438 from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:42501: session id = 8559229e044b4e1ea661b881b3f9af50-718853c0da3d4a68ab478a6e79cdd438
I20240501 20:10:22.721248 21248 tablet_copy_source_session.cc:205] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.723402 30279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718853c0da3d4a68ab478a6e79cdd438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.723443 30216 tablet_copy_client.cc:523] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:22.730393 30216 tablet_bootstrap.cc:492] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:22.734320 30254 tablet_bootstrap.cc:492] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:22.734841 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:22.744019 30279 tablet_copy_client.cc:783] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:22.744814 30279 tablet_copy_client.cc:647] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:22.750516 30279 tablet_copy_client.cc:523] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:22.755373 30279 tablet_bootstrap.cc:492] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:22.773023 30216 tablet_bootstrap.cc:492] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:22.773488 30216 tablet_bootstrap.cc:492] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:10:22.773778 30216 ts_tablet_manager.cc:1367] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.043s user 0.020s sys 0.008s
I20240501 20:10:22.774744 30216 raft_consensus.cc:348] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:22.775049 30216 raft_consensus.cc:729] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.775275 30216 consensus_queue.cc:260] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:22.789634 30216 ts_tablet_manager.cc:1398] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240501 20:10:22.790431 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-6c31dffb0482438ebbc6243ad54752e9 received from {username='slave'} at 127.13.251.130:58461
I20240501 20:10:22.790544 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-6c31dffb0482438ebbc6243ad54752e9 on tablet 6c31dffb0482438ebbc6243ad54752e9 with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:10:22.818764 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:22.823148 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:22.830927 30254 tablet_bootstrap.cc:492] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:22.831399 30254 tablet_bootstrap.cc:492] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap complete.
I20240501 20:10:22.831676 30254 ts_tablet_manager.cc:1367] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.097s user 0.006s sys 0.001s
I20240501 20:10:22.832374 30254 raft_consensus.cc:348] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:22.832618 30254 raft_consensus.cc:729] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.832850 30254 consensus_queue.cc:260] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:22.833458 30254 ts_tablet_manager.cc:1398] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:22.839054 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-8978a4c14f0a4446a979d3fc05248085 received from {username='slave'} at 127.13.251.134:60949
I20240501 20:10:22.839175 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-8978a4c14f0a4446a979d3fc05248085 on tablet 8978a4c14f0a4446a979d3fc05248085 with peer c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:10:22.854275 30279 tablet_bootstrap.cc:492] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:22.854717 30279 tablet_bootstrap.cc:492] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:10:22.854970 30279 ts_tablet_manager.cc:1367] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.100s user 0.029s sys 0.018s
I20240501 20:10:22.855751 30279 raft_consensus.cc:348] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:22.855952 30279 raft_consensus.cc:729] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.856142 30279 consensus_queue.cc:260] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:22.861461 30279 ts_tablet_manager.cc:1398] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:10:22.868366 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-718853c0da3d4a68ab478a6e79cdd438 received from {username='slave'} at 127.13.251.136:42501
I20240501 20:10:22.868486 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-718853c0da3d4a68ab478a6e79cdd438 on tablet 718853c0da3d4a68ab478a6e79cdd438 with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:10:22.898877 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:22.924826 21097 raft_consensus.cc:1203] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:22.934656 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:22.944007 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:22.955006 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:22.986759 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:22.998580 23846 raft_consensus.cc:1203] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:23.089324 24109 raft_consensus.cc:1203] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:23.104724 23843 raft_consensus.cc:1203] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:23.135322 21095 raft_consensus.cc:1203] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:23.190045 21358 raft_consensus.cc:1203] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:23.457046 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:23.513571 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:23.518438 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:23.529438 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:23.560451 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:23.572989 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:23.660985 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:23.668967 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:23.769415 23709 consensus_queue.cc:237] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:23.771523 21356 raft_consensus.cc:1261] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:23.771523 23974 raft_consensus.cc:1261] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:23.772058 30368 consensus_queue.cc:1035] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:23.772305 30120 consensus_queue.cc:1035] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:23.775139 30368 raft_consensus.cc:2932] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:23.775808 23977 raft_consensus.cc:2932] T 83384d743f874d77bc8a12d1c9834736 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:23.775782 21356 raft_consensus.cc:2932] T 83384d743f874d77bc8a12d1c9834736 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:23.783486 20817 catalog_manager.cc:5484] T 83384d743f874d77bc8a12d1c9834736 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:23.788946 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 83384d743f874d77bc8a12d1c9834736 with delete_type TABLET_DATA_TOMBSTONED (TS cda19d6a81c0473db302535ce2ada7f0 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54238
I20240501 20:10:23.797328 30406 tablet_replica.cc:330] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:10:23.797555 30406 raft_consensus.cc:2229] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:23.798031 30406 raft_consensus.cc:2258] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:23.798614 30406 ts_tablet_manager.cc:1875] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:23.803831 30406 ts_tablet_manager.cc:1888] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:23.804061 30406 log.cc:1196] T 83384d743f874d77bc8a12d1c9834736 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/83384d743f874d77bc8a12d1c9834736
I20240501 20:10:23.805995 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 83384d743f874d77bc8a12d1c9834736 (table rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]) successfully deleted
I20240501 20:10:24.220019 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:10:24.294070 24043 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 61514.4us
I20240501 20:10:24.311597 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:24.364864 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:24.367873 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:24.369874 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:24.376724 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:24.378350 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:24.389155 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:24.596931 21227 consensus_queue.cc:237] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:24.598698 20962 raft_consensus.cc:1261] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.598845 24110 raft_consensus.cc:1261] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.599210 30483 consensus_queue.cc:1035] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.599474 30159 consensus_queue.cc:1035] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.601864 30483 raft_consensus.cc:2932] T 718853c0da3d4a68ab478a6e79cdd438 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:24.602392 24110 raft_consensus.cc:2932] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:24.602881 20962 raft_consensus.cc:2932] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:24.607847 20818 catalog_manager.cc:5484] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:24.617928 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 718853c0da3d4a68ab478a6e79cdd438 with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:24.618664 30506 tablet_replica.cc:330] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:24.618896 30506 raft_consensus.cc:2229] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:24.619148 30506 raft_consensus.cc:2258] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:24.619829 30506 ts_tablet_manager.cc:1875] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:24.627760 30506 ts_tablet_manager.cc:1888] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:24.627909 30506 log.cc:1196] T 718853c0da3d4a68ab478a6e79cdd438 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/718853c0da3d4a68ab478a6e79cdd438
I20240501 20:10:24.628866 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 718853c0da3d4a68ab478a6e79cdd438 (table rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]) successfully deleted
I20240501 20:10:25.171355 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:25.290889 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:25.363252 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:25.432806 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:25.439397 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:25.442976 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:25.444505 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:25.448107 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:25.641100 23709 consensus_queue.cc:237] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:25.643289 21095 raft_consensus.cc:1261] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:25.643432 21228 raft_consensus.cc:1261] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:25.643832 30120 consensus_queue.cc:1035] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:25.644063 30368 consensus_queue.cc:1035] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:25.646510 30120 raft_consensus.cc:2932] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:25.649734 21228 raft_consensus.cc:2932] T 0fda032f2be744fe95170b3c3287d6ae P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:25.653963 20817 catalog_manager.cc:5484] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:25.654233 21095 raft_consensus.cc:2932] T 0fda032f2be744fe95170b3c3287d6ae P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:25.673367 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fda032f2be744fe95170b3c3287d6ae with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:25.679665 30600 tablet_replica.cc:330] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:25.679994 30600 raft_consensus.cc:2229] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:25.680234 30600 raft_consensus.cc:2258] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:25.681090 30600 ts_tablet_manager.cc:1875] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:25.689421 30600 ts_tablet_manager.cc:1888] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:25.689509 30600 log.cc:1196] T 0fda032f2be744fe95170b3c3287d6ae P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/0fda032f2be744fe95170b3c3287d6ae
I20240501 20:10:25.690451 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 0fda032f2be744fe95170b3c3287d6ae (table rebalancer_extra_table_12 [id=f38ed2928c8c41fdbec296326253d835]) successfully deleted
I20240501 20:10:25.912744 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:25.918813 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:25.942148 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:25.944749 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:26.000811 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:26.047890 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:26.059509 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:26.161542 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:26.291010 23710 consensus_queue.cc:237] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:26.293311 21095 raft_consensus.cc:1261] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.293854 23974 raft_consensus.cc:1261] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.294148 30120 consensus_queue.cc:1035] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.294418 30677 consensus_queue.cc:1035] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.297617 30598 raft_consensus.cc:2932] T 25dee912ab5f4613b66d056a8731f188 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:26.298132 23974 raft_consensus.cc:2932] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:26.299955 21095 raft_consensus.cc:2932] T 25dee912ab5f4613b66d056a8731f188 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:26.305405 20826 catalog_manager.cc:5484] T 25dee912ab5f4613b66d056a8731f188 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:26.314554 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 25dee912ab5f4613b66d056a8731f188 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:10:26.315310 30688 tablet_replica.cc:330] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:10:26.315800 30688 raft_consensus.cc:2229] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.316211 30688 raft_consensus.cc:2258] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.316932 30688 ts_tablet_manager.cc:1875] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:26.324254 30688 ts_tablet_manager.cc:1888] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:26.324456 30688 log.cc:1196] T 25dee912ab5f4613b66d056a8731f188 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/25dee912ab5f4613b66d056a8731f188
I20240501 20:10:26.325423 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 25dee912ab5f4613b66d056a8731f188 (table rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]) successfully deleted
I20240501 20:10:26.597124 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:26.624441 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:26.653589 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:26.654350 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:26.664947 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:26.724854 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:26.729110 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:26.769758 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:26.892887 21097 consensus_queue.cc:237] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:26.895582 23710 raft_consensus.cc:1261] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:26.895809 21227 raft_consensus.cc:1261] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:26.896121 30573 consensus_queue.cc:1035] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:26.896344 30482 consensus_queue.cc:1035] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:26.898852 30573 raft_consensus.cc:2932] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:26.899870 23710 raft_consensus.cc:2932] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:26.899890 21227 raft_consensus.cc:2932] T c58dd83ebd0949dea6f4567067c8dfc9 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:26.907352 20817 catalog_manager.cc:5484] T c58dd83ebd0949dea6f4567067c8dfc9 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 2 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:26.930413 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet c58dd83ebd0949dea6f4567067c8dfc9 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57076
I20240501 20:10:26.931013 30779 tablet_replica.cc:330] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:10:26.931279 30779 raft_consensus.cc:2229] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.931470 30779 raft_consensus.cc:2258] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.932235 30779 ts_tablet_manager.cc:1875] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:26.940531 30779 ts_tablet_manager.cc:1888] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 20:10:26.940732 30779 log.cc:1196] T c58dd83ebd0949dea6f4567067c8dfc9 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/c58dd83ebd0949dea6f4567067c8dfc9
I20240501 20:10:26.941763 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet c58dd83ebd0949dea6f4567067c8dfc9 (table rebalancer_extra_table_50 [id=1b81e44719b44272a1f3de54f6eea75f]) successfully deleted
I20240501 20:10:27.351984 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:27.361446 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:27.362135 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:27.371124 21207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:27.400154 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:27.413283 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:27.518580 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:27.604895 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:27.719856 21227 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8978a4c14f0a4446a979d3fc05248085"
dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52964
I20240501 20:10:27.720062 21227 raft_consensus.cc:595] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:27.891278 30658 raft_consensus.cc:979] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: : Instructing follower 8559229e044b4e1ea661b881b3f9af50 to start an election
I20240501 20:10:27.891469 30658 raft_consensus.cc:1067] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Signalling peer 8559229e044b4e1ea661b881b3f9af50 to start an election
I20240501 20:10:27.892355 24108 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8978a4c14f0a4446a979d3fc05248085"
dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
from {username='slave'} at 127.13.251.131:53727
I20240501 20:10:27.892540 24108 raft_consensus.cc:482] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:27.892623 24108 raft_consensus.cc:3037] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:27.897451 24108 raft_consensus.cc:504] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:27.899237 24108 leader_election.cc:288] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:27.899493 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8978a4c14f0a4446a979d3fc05248085" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:10:27.899714 21227 raft_consensus.cc:3032] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:27.899794 21227 raft_consensus.cc:729] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Running, Role: LEADER
I20240501 20:10:27.900043 21227 consensus_queue.cc:260] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:27.900524 21227 raft_consensus.cc:3037] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:27.903378 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8978a4c14f0a4446a979d3fc05248085" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:10:27.903579 23977 raft_consensus.cc:3037] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:27.905920 23977 raft_consensus.cc:2454] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 2.
I20240501 20:10:27.910521 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8978a4c14f0a4446a979d3fc05248085" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:10:27.910707 23846 raft_consensus.cc:3037] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:27.913250 23846 raft_consensus.cc:2454] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 2.
I20240501 20:10:27.923418 24046 leader_election.cc:302] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, 8559229e044b4e1ea661b881b3f9af50, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:10:27.923804 30572 raft_consensus.cc:2781] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:27.903131 21227 raft_consensus.cc:2454] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 2.
I20240501 20:10:27.967358 30572 raft_consensus.cc:686] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 2 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:10:27.974085 30572 consensus_queue.cc:237] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:28.005599 20823 catalog_manager.cc:5484] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 2 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:28.058869 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:28.150420 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:28.185349 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:28.191570 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:28.213004 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:28.218396 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:28.264025 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:28.275300 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:28.435724 23846 raft_consensus.cc:1261] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:28.436368 30572 consensus_queue.cc:1035] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.459564 21227 raft_consensus.cc:1261] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:28.461031 30572 consensus_queue.cc:1035] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.483505 23975 raft_consensus.cc:1261] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:28.484148 30572 consensus_queue.cc:1035] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.818908 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:28.849385 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:28.869441 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:28.886540 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:28.930011 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:28.950613 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:28.993618 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:29.079054 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:29.193817 20963 consensus_queue.cc:237] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:29.195931 23846 raft_consensus.cc:1261] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:29.196293 21358 raft_consensus.cc:1261] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:29.196494 30585 consensus_queue.cc:1035] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:29.198127 30585 consensus_queue.cc:1035] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:29.200309 30585 raft_consensus.cc:2932] T b7eb4c409a2945f0b39bebd37fec4ea7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:29.200752 23846 raft_consensus.cc:2932] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:29.207598 20821 catalog_manager.cc:5484] T b7eb4c409a2945f0b39bebd37fec4ea7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:29.218067 21358 raft_consensus.cc:2932] T b7eb4c409a2945f0b39bebd37fec4ea7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:29.230289 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet b7eb4c409a2945f0b39bebd37fec4ea7 with delete_type TABLET_DATA_TOMBSTONED (TS 4af096f5a6a9440a94f6682b399efe20 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45322
I20240501 20:10:29.230917 31046 tablet_replica.cc:330] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:10:29.231277 31046 raft_consensus.cc:2229] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:29.231570 31046 raft_consensus.cc:2258] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:29.232316 31046 ts_tablet_manager.cc:1875] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:29.240778 31046 ts_tablet_manager.cc:1888] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:29.241029 31046 log.cc:1196] T b7eb4c409a2945f0b39bebd37fec4ea7 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/b7eb4c409a2945f0b39bebd37fec4ea7
I20240501 20:10:29.242012 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet b7eb4c409a2945f0b39bebd37fec4ea7 (table rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]) successfully deleted
I20240501 20:10:29.281399 21358 consensus_queue.cc:237] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:29.284067 24109 raft_consensus.cc:1261] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.284399 23975 raft_consensus.cc:1261] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.284749 30378 consensus_queue.cc:1035] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:29.285086 21291 consensus_peers.cc:485] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25c9aac0679a41579f390c49f546f66b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.285434 30378 consensus_queue.cc:1035] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.291532 30275 raft_consensus.cc:2932] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:29.292503 23975 raft_consensus.cc:2932] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:29.297679 24109 raft_consensus.cc:2932] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:29.299705 20818 catalog_manager.cc:5484] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:29.338641 21095 consensus_queue.cc:237] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:29.341454 23709 raft_consensus.cc:1261] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.341590 24109 raft_consensus.cc:1261] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:29.342222 21031 consensus_peers.cc:485] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Couldn't send request to peer 744c5a532e724391830c0785d32eb2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20749a5ab2744578afd2978320e7fbf3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.342334 30573 consensus_queue.cc:1035] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.342631 30936 consensus_queue.cc:1035] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.345261 30573 raft_consensus.cc:2932] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:29.347368 23709 raft_consensus.cc:2932] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:29.349782 24109 raft_consensus.cc:2932] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:29.355368 20826 catalog_manager.cc:5484] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index -1 to 2, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:29.398042 21095 consensus_queue.cc:237] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:29.400638 24109 raft_consensus.cc:1261] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.400668 21227 raft_consensus.cc:1261] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:29.400718 21032 consensus_peers.cc:485] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d308589b223471bb2f20ff3c6a7f13e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.401240 30936 consensus_queue.cc:1035] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.401530 31085 consensus_queue.cc:1035] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.404366 30936 raft_consensus.cc:2932] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:29.404907 24109 raft_consensus.cc:2932] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:29.407732 21227 raft_consensus.cc:2932] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:29.410756 20818 catalog_manager.cc:5484] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:29.453262 21227 consensus_queue.cc:237] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
W20240501 20:10:29.455708 21161 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b472171e77e144ffa8fa4cdbd40a03a0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.455833 23843 raft_consensus.cc:1261] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.456303 24108 raft_consensus.cc:1261] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.456565 31105 consensus_queue.cc:1035] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.457610 31105 consensus_queue.cc:1035] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.460076 30658 raft_consensus.cc:2932] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:29.460615 24108 raft_consensus.cc:2932] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:29.469383 20821 catalog_manager.cc:5484] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:29.472173 23843 raft_consensus.cc:2932] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:29.566900 23843 consensus_queue.cc:237] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:29.578722 20963 raft_consensus.cc:1261] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:29.579146 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0722c64801734730abefe2a742da549e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.579526 30442 consensus_queue.cc:1035] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.592518 23709 raft_consensus.cc:1261] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.593158 30195 consensus_queue.cc:1035] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.602568 31143 raft_consensus.cc:2932] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:29.603318 20963 raft_consensus.cc:2932] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:29.607882 23709 raft_consensus.cc:2932] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:29.614391 20821 catalog_manager.cc:5484] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:29.654541 21095 consensus_queue.cc:237] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:29.656877 21227 raft_consensus.cc:1261] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:29.657104 21031 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Couldn't send request to peer 744c5a532e724391830c0785d32eb2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a7e956aedc44f56b3bf629c80ff65e4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.657230 21358 raft_consensus.cc:1261] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:29.657778 30573 consensus_queue.cc:1035] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:29.658146 30936 consensus_queue.cc:1035] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:29.661540 31082 raft_consensus.cc:2932] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:29.662173 21227 raft_consensus.cc:2932] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:29.665398 21358 raft_consensus.cc:2932] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:29.668398 20826 catalog_manager.cc:5484] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:29.706491 23977 consensus_queue.cc:237] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:29.709648 24108 raft_consensus.cc:1261] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.710786 23843 raft_consensus.cc:1261] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:29.710796 23911 consensus_peers.cc:485] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88f0ef64f05640c1a9e58e528149f8d0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.711331 31182 consensus_queue.cc:1035] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.711597 31175 consensus_queue.cc:1035] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.717085 31175 raft_consensus.cc:2932] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:29.717885 24108 raft_consensus.cc:2932] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:29.723978 20821 catalog_manager.cc:5484] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:29.724771 23843 raft_consensus.cc:2932] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:29.737607 31187 ts_tablet_manager.cc:902] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:29.761250 31195 ts_tablet_manager.cc:902] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:10:29.762640 31187 tablet_copy_client.cc:308] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:10:29.763382 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 25c9aac0679a41579f390c49f546f66b from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:57815)
I20240501 20:10:29.763557 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 25c9aac0679a41579f390c49f546f66b from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:57815: session id = cda19d6a81c0473db302535ce2ada7f0-25c9aac0679a41579f390c49f546f66b
I20240501 20:10:29.766376 21378 tablet_copy_source_session.cc:205] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:29.768060 31187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c9aac0679a41579f390c49f546f66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.775328 31195 tablet_copy_client.cc:308] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:10:29.776178 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet 20749a5ab2744578afd2978320e7fbf3 from peer 744c5a532e724391830c0785d32eb2fb ({username='slave'} at 127.13.251.135:53413)
I20240501 20:10:29.776350 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet 20749a5ab2744578afd2978320e7fbf3 from peer 744c5a532e724391830c0785d32eb2fb at {username='slave'} at 127.13.251.135:53413: session id = 744c5a532e724391830c0785d32eb2fb-20749a5ab2744578afd2978320e7fbf3
I20240501 20:10:29.777400 31187 tablet_copy_client.cc:783] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:29.777679 31187 tablet_copy_client.cc:647] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:29.781169 21118 tablet_copy_source_session.cc:205] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:29.783247 31187 tablet_copy_client.cc:523] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:29.784180 31195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20749a5ab2744578afd2978320e7fbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.787223 31187 tablet_bootstrap.cc:492] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:29.790340 31195 tablet_copy_client.cc:783] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:29.790728 31195 tablet_copy_client.cc:647] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:29.794101 31195 tablet_copy_client.cc:523] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:29.798928 31195 tablet_bootstrap.cc:492] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:29.867571 31211 ts_tablet_manager.cc:902] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
W20240501 20:10:29.868566 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 45640.8us
I20240501 20:10:29.879817 31211 tablet_copy_client.cc:308] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:10:29.880669 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet b472171e77e144ffa8fa4cdbd40a03a0 from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:59479)
I20240501 20:10:29.880859 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet b472171e77e144ffa8fa4cdbd40a03a0 from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:59479: session id = cda19d6a81c0473db302535ce2ada7f0-b472171e77e144ffa8fa4cdbd40a03a0
I20240501 20:10:29.884403 31206 ts_tablet_manager.cc:902] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:10:29.885735 21248 tablet_copy_source_session.cc:205] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:29.888943 31211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b472171e77e144ffa8fa4cdbd40a03a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.889014 31187 tablet_bootstrap.cc:492] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:29.889487 31187 tablet_bootstrap.cc:492] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:10:29.889753 31187 ts_tablet_manager.cc:1367] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.103s user 0.044s sys 0.049s
I20240501 20:10:29.890619 31187 raft_consensus.cc:348] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:29.890909 31187 raft_consensus.cc:729] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.891219 31187 consensus_queue.cc:260] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:29.894091 31187 ts_tablet_manager.cc:1398] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:10:29.895107 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-25c9aac0679a41579f390c49f546f66b received from {username='slave'} at 127.13.251.129:57815
I20240501 20:10:29.895221 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-25c9aac0679a41579f390c49f546f66b on tablet 25c9aac0679a41579f390c49f546f66b with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:29.895576 31195 tablet_bootstrap.cc:492] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:29.895931 31195 tablet_bootstrap.cc:492] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: Bootstrap complete.
I20240501 20:10:29.896220 31195 ts_tablet_manager.cc:1367] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.097s user 0.020s sys 0.027s
I20240501 20:10:29.896947 31195 raft_consensus.cc:348] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:29.897190 31195 raft_consensus.cc:729] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.897442 31195 consensus_queue.cc:260] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:29.899006 31195 ts_tablet_manager.cc:1398] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:29.900847 31211 tablet_copy_client.cc:783] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:29.900842 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session 744c5a532e724391830c0785d32eb2fb-20749a5ab2744578afd2978320e7fbf3 received from {username='slave'} at 127.13.251.135:53413
I20240501 20:10:29.900964 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session 744c5a532e724391830c0785d32eb2fb-20749a5ab2744578afd2978320e7fbf3 on tablet 20749a5ab2744578afd2978320e7fbf3 with peer 744c5a532e724391830c0785d32eb2fb
I20240501 20:10:29.901101 31211 tablet_copy_client.cc:647] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:29.903425 31211 tablet_copy_client.cc:523] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:29.906368 31211 tablet_bootstrap.cc:492] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:29.912216 31206 tablet_copy_client.cc:308] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:10:29.913174 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet 9d308589b223471bb2f20ff3c6a7f13e from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:47489)
I20240501 20:10:29.913350 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet 9d308589b223471bb2f20ff3c6a7f13e from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:47489: session id = 61380023001a4ee9ada9ec2c6c1b7159-9d308589b223471bb2f20ff3c6a7f13e
I20240501 20:10:29.914729 31211 tablet_bootstrap.cc:492] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:29.915035 31211 tablet_bootstrap.cc:492] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:10:29.915252 31211 ts_tablet_manager.cc:1367] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 20:10:29.915855 31211 raft_consensus.cc:348] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:29.916004 21118 tablet_copy_source_session.cc:205] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:29.916023 31211 raft_consensus.cc:729] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.916229 31211 consensus_queue.cc:260] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:29.916790 31211 ts_tablet_manager.cc:1398] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 20:10:29.918011 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-b472171e77e144ffa8fa4cdbd40a03a0 received from {username='slave'} at 127.13.251.129:59479
I20240501 20:10:29.918133 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-b472171e77e144ffa8fa4cdbd40a03a0 on tablet b472171e77e144ffa8fa4cdbd40a03a0 with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:29.921288 31206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d308589b223471bb2f20ff3c6a7f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.929263 31206 tablet_copy_client.cc:783] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:29.929744 31206 tablet_copy_client.cc:647] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:29.931895 31206 tablet_copy_client.cc:523] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:29.935312 31206 tablet_bootstrap.cc:492] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:29.935513 21228 consensus_queue.cc:237] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
W20240501 20:10:29.938163 21164 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 -> Peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Couldn't send request to peer c6dcaa8efff642dcb7c2c6e8be3a65f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 021fed14d5a140f3b42d359bae264f69. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.939155 20963 raft_consensus.cc:1261] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:29.939787 31105 consensus_queue.cc:1035] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:29.942226 21095 raft_consensus.cc:1261] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:29.942835 30658 consensus_queue.cc:1035] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:29.946465 31206 tablet_bootstrap.cc:492] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:29.946856 31206 tablet_bootstrap.cc:492] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:10:29.947113 31206 ts_tablet_manager.cc:1367] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.012s user 0.014s sys 0.000s
I20240501 20:10:29.947057 31105 raft_consensus.cc:2932] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:29.947878 31206 raft_consensus.cc:348] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:29.948187 31206 raft_consensus.cc:729] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.948464 31206 consensus_queue.cc:260] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:29.949065 31206 ts_tablet_manager.cc:1398] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:29.951603 21095 raft_consensus.cc:2932] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:29.952809 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-9d308589b223471bb2f20ff3c6a7f13e received from {username='slave'} at 127.13.251.132:47489
I20240501 20:10:29.952939 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-9d308589b223471bb2f20ff3c6a7f13e on tablet 9d308589b223471bb2f20ff3c6a7f13e with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:10:29.961365 20963 raft_consensus.cc:2932] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:29.959990 20818 catalog_manager.cc:5484] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:29.985221 31238 ts_tablet_manager.cc:902] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:30.012724 31238 tablet_copy_client.cc:308] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:10:30.016187 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet 0722c64801734730abefe2a742da549e from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:54857)
I20240501 20:10:30.016342 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet 0722c64801734730abefe2a742da549e from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:54857: session id = 4af096f5a6a9440a94f6682b399efe20-0722c64801734730abefe2a742da549e
I20240501 20:10:30.019574 23866 tablet_copy_source_session.cc:205] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.024070 31238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0722c64801734730abefe2a742da549e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.038661 31238 tablet_copy_client.cc:783] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.046856 31238 tablet_copy_client.cc:647] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.048859 31238 tablet_copy_client.cc:523] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.059046 31238 tablet_bootstrap.cc:492] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:30.068886 20963 consensus_queue.cc:237] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:30.073158 23977 raft_consensus.cc:1261] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:30.074240 20901 consensus_peers.cc:485] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca4fe4379c854ff1a0b8f9f17a6311cf. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.074669 31219 consensus_queue.cc:1035] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:30.075080 31238 tablet_bootstrap.cc:492] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:30.075439 31238 tablet_bootstrap.cc:492] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:10:30.075769 31238 ts_tablet_manager.cc:1367] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240501 20:10:30.076043 21228 raft_consensus.cc:1261] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:30.076560 31219 consensus_queue.cc:1035] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:30.076750 31238 raft_consensus.cc:348] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:30.077005 31238 raft_consensus.cc:729] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:30.077313 31238 consensus_queue.cc:260] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:30.079449 31238 ts_tablet_manager.cc:1398] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:10:30.083684 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-0722c64801734730abefe2a742da549e received from {username='slave'} at 127.13.251.131:54857
I20240501 20:10:30.083803 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-0722c64801734730abefe2a742da549e on tablet 0722c64801734730abefe2a742da549e with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:10:30.107637 30481 raft_consensus.cc:2932] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:30.108450 21228 raft_consensus.cc:2932] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:30.120409 20821 catalog_manager.cc:5484] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:30.129240 23977 raft_consensus.cc:2932] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:30.163657 21356 consensus_queue.cc:237] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:30.166209 21095 raft_consensus.cc:1261] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:30.166271 21228 raft_consensus.cc:1261] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:30.166492 21293 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b4384496d9048259bf353bd58a15008. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.166833 30275 consensus_queue.cc:1035] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:30.167129 31210 consensus_queue.cc:1035] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:30.170028 31210 raft_consensus.cc:2932] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:30.171833 21095 raft_consensus.cc:2932] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:30.174443 21228 raft_consensus.cc:2932] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:30.180017 20818 catalog_manager.cc:5484] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:30.185968 21356 raft_consensus.cc:1203] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:30.214272 31287 ts_tablet_manager.cc:902] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:30.215481 31287 tablet_copy_client.cc:308] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:10:30.216140 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet 88f0ef64f05640c1a9e58e528149f8d0 from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:56741)
I20240501 20:10:30.216296 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet 88f0ef64f05640c1a9e58e528149f8d0 from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:56741: session id = 51bf640e895b4e5eb91bdf7509ea7e46-88f0ef64f05640c1a9e58e528149f8d0
I20240501 20:10:30.218657 23997 tablet_copy_source_session.cc:205] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.219730 31287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f0ef64f05640c1a9e58e528149f8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.225855 31287 tablet_copy_client.cc:783] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.226089 31287 tablet_copy_client.cc:647] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.229032 23977 raft_consensus.cc:1203] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:30.233614 31287 tablet_copy_client.cc:523] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.236510 31287 tablet_bootstrap.cc:492] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:30.241782 31287 tablet_bootstrap.cc:492] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:30.242307 31287 tablet_bootstrap.cc:492] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:10:30.242568 31287 ts_tablet_manager.cc:1367] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:10:30.243127 31287 raft_consensus.cc:348] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:30.243304 31287 raft_consensus.cc:729] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:30.243613 31287 consensus_queue.cc:260] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:30.244303 31287 ts_tablet_manager.cc:1398] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:30.245298 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-88f0ef64f05640c1a9e58e528149f8d0 received from {username='slave'} at 127.13.251.133:56741
I20240501 20:10:30.245407 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-88f0ef64f05640c1a9e58e528149f8d0 on tablet 88f0ef64f05640c1a9e58e528149f8d0 with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:10:30.250042 24108 consensus_queue.cc:237] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:30.252475 21228 raft_consensus.cc:1261] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:30.252584 23977 raft_consensus.cc:1261] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:30.253196 31249 consensus_queue.cc:1035] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:30.253490 31087 consensus_queue.cc:1035] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:30.256667 31195 ts_tablet_manager.cc:902] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
W20240501 20:10:30.260876 24043 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3c0aa45580749a4bd6fa7ccd635cb38. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.280818 20963 raft_consensus.cc:1203] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:30.286862 31195 tablet_copy_client.cc:308] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:10:30.287555 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet 3a7e956aedc44f56b3bf629c80ff65e4 from peer 744c5a532e724391830c0785d32eb2fb ({username='slave'} at 127.13.251.135:53413)
I20240501 20:10:30.287740 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet 3a7e956aedc44f56b3bf629c80ff65e4 from peer 744c5a532e724391830c0785d32eb2fb at {username='slave'} at 127.13.251.135:53413: session id = 744c5a532e724391830c0785d32eb2fb-3a7e956aedc44f56b3bf629c80ff65e4
I20240501 20:10:30.288810 31299 raft_consensus.cc:2932] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:30.289551 20963 raft_consensus.cc:1203] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:30.290588 21118 tablet_copy_source_session.cc:205] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.290488 21228 raft_consensus.cc:2932] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:30.292372 31195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7e956aedc44f56b3bf629c80ff65e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.293923 23977 raft_consensus.cc:2932] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:30.300670 20821 catalog_manager.cc:5484] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:30.302285 31195 tablet_copy_client.cc:783] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.302541 31195 tablet_copy_client.cc:647] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.308070 31195 tablet_copy_client.cc:523] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.310943 31195 tablet_bootstrap.cc:492] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:30.350091 31195 tablet_bootstrap.cc:492] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:30.350672 31195 tablet_bootstrap.cc:492] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: Bootstrap complete.
I20240501 20:10:30.350966 31195 ts_tablet_manager.cc:1367] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.040s user 0.039s sys 0.000s
I20240501 20:10:30.351615 31195 raft_consensus.cc:348] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:30.351794 31195 raft_consensus.cc:729] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:30.351991 31195 consensus_queue.cc:260] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:30.353765 31195 ts_tablet_manager.cc:1398] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:30.354419 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session 744c5a532e724391830c0785d32eb2fb-3a7e956aedc44f56b3bf629c80ff65e4 received from {username='slave'} at 127.13.251.135:53413
I20240501 20:10:30.354553 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session 744c5a532e724391830c0785d32eb2fb-3a7e956aedc44f56b3bf629c80ff65e4 on tablet 3a7e956aedc44f56b3bf629c80ff65e4 with peer 744c5a532e724391830c0785d32eb2fb
I20240501 20:10:30.413995 31336 ts_tablet_manager.cc:902] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:30.427734 31336 tablet_copy_client.cc:308] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:10:30.430534 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet 021fed14d5a140f3b42d359bae264f69 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 ({username='slave'} at 127.13.251.134:60949)
I20240501 20:10:30.430682 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet 021fed14d5a140f3b42d359bae264f69 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 at {username='slave'} at 127.13.251.134:60949: session id = c6dcaa8efff642dcb7c2c6e8be3a65f7-021fed14d5a140f3b42d359bae264f69
I20240501 20:10:30.433490 21248 tablet_copy_source_session.cc:205] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.437940 31336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021fed14d5a140f3b42d359bae264f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.442497 31336 tablet_copy_client.cc:783] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.442704 31336 tablet_copy_client.cc:647] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.447067 31336 tablet_copy_client.cc:523] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.453419 31336 tablet_bootstrap.cc:492] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:30.558207 31336 tablet_bootstrap.cc:492] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:30.558619 31336 tablet_bootstrap.cc:492] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap complete.
I20240501 20:10:30.558929 31336 ts_tablet_manager.cc:1367] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.106s user 0.072s sys 0.017s
I20240501 20:10:30.559701 31336 raft_consensus.cc:348] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:30.559893 31336 raft_consensus.cc:729] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:30.560150 31336 consensus_queue.cc:260] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:30.560662 31336 ts_tablet_manager.cc:1398] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:30.563570 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-021fed14d5a140f3b42d359bae264f69 received from {username='slave'} at 127.13.251.134:60949
I20240501 20:10:30.563694 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-021fed14d5a140f3b42d359bae264f69 on tablet 021fed14d5a140f3b42d359bae264f69 with peer c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:10:30.596643 21228 raft_consensus.cc:1203] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:30.635636 31354 ts_tablet_manager.cc:902] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:30.655975 31354 tablet_copy_client.cc:308] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:41259
I20240501 20:10:30.656810 20987 tablet_copy_service.cc:140] P cda19d6a81c0473db302535ce2ada7f0: Received BeginTabletCopySession request for tablet ca4fe4379c854ff1a0b8f9f17a6311cf from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:44313)
I20240501 20:10:30.657011 20987 tablet_copy_service.cc:161] P cda19d6a81c0473db302535ce2ada7f0: Beginning new tablet copy session on tablet ca4fe4379c854ff1a0b8f9f17a6311cf from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:44313: session id = 61380023001a4ee9ada9ec2c6c1b7159-ca4fe4379c854ff1a0b8f9f17a6311cf
I20240501 20:10:30.661747 20987 tablet_copy_source_session.cc:205] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.663621 31354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4fe4379c854ff1a0b8f9f17a6311cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.675238 31354 tablet_copy_client.cc:783] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.675623 31354 tablet_copy_client.cc:647] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.677502 31354 tablet_copy_client.cc:523] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.681818 31354 tablet_bootstrap.cc:492] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:30.692389 31354 tablet_bootstrap.cc:492] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:30.692890 31354 tablet_bootstrap.cc:492] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:10:30.693414 31354 ts_tablet_manager.cc:1367] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240501 20:10:30.694090 31354 raft_consensus.cc:348] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:30.694276 31354 raft_consensus.cc:729] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:30.694483 31354 consensus_queue.cc:260] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:30.695055 31354 ts_tablet_manager.cc:1398] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:30.696483 20987 tablet_copy_service.cc:342] P cda19d6a81c0473db302535ce2ada7f0: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-ca4fe4379c854ff1a0b8f9f17a6311cf received from {username='slave'} at 127.13.251.132:44313
I20240501 20:10:30.696592 20987 tablet_copy_service.cc:433] P cda19d6a81c0473db302535ce2ada7f0: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-ca4fe4379c854ff1a0b8f9f17a6311cf on tablet ca4fe4379c854ff1a0b8f9f17a6311cf with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:10:30.714704 31367 ts_tablet_manager.cc:902] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:30.716171 31367 tablet_copy_client.cc:308] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:10:30.716912 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 2b4384496d9048259bf353bd58a15008 from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:58619)
I20240501 20:10:30.717087 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 2b4384496d9048259bf353bd58a15008 from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:58619: session id = 8559229e044b4e1ea661b881b3f9af50-2b4384496d9048259bf353bd58a15008
I20240501 20:10:30.720027 21378 tablet_copy_source_session.cc:205] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.723104 31367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4384496d9048259bf353bd58a15008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.740986 31367 tablet_copy_client.cc:783] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.741416 31367 tablet_copy_client.cc:647] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.753284 31378 ts_tablet_manager.cc:902] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:30.757262 31367 tablet_copy_client.cc:523] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.764029 23709 raft_consensus.cc:1203] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:30.768410 31367 tablet_bootstrap.cc:492] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:30.782590 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:30.795102 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:30.805961 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:30.827103 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:30.841401 31378 tablet_copy_client.cc:308] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.136:44267
I20240501 20:10:30.842219 24130 tablet_copy_service.cc:140] P 8559229e044b4e1ea661b881b3f9af50: Received BeginTabletCopySession request for tablet d3c0aa45580749a4bd6fa7ccd635cb38 from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:53433)
I20240501 20:10:30.842407 24130 tablet_copy_service.cc:161] P 8559229e044b4e1ea661b881b3f9af50: Beginning new tablet copy session on tablet d3c0aa45580749a4bd6fa7ccd635cb38 from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:53433: session id = cda19d6a81c0473db302535ce2ada7f0-d3c0aa45580749a4bd6fa7ccd635cb38
I20240501 20:10:30.845161 24130 tablet_copy_source_session.cc:205] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.846736 31378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c0aa45580749a4bd6fa7ccd635cb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.864562 31378 tablet_copy_client.cc:783] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.864857 31378 tablet_copy_client.cc:647] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.872001 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:30.872054 31378 tablet_copy_client.cc:523] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.880527 23977 raft_consensus.cc:1203] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:30.960072 31367 tablet_bootstrap.cc:492] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:30.960602 31367 tablet_bootstrap.cc:492] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:10:30.960906 31367 ts_tablet_manager.cc:1367] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.193s user 0.072s sys 0.020s
I20240501 20:10:30.961869 31367 raft_consensus.cc:348] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:30.962143 31367 raft_consensus.cc:729] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:30.964478 31367 consensus_queue.cc:260] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:30.965222 31367 ts_tablet_manager.cc:1398] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:30.966120 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-2b4384496d9048259bf353bd58a15008 received from {username='slave'} at 127.13.251.136:58619
I20240501 20:10:30.966238 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-2b4384496d9048259bf353bd58a15008 on tablet 2b4384496d9048259bf353bd58a15008 with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:10:30.978932 23843 raft_consensus.cc:1203] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:30.996959 31378 tablet_bootstrap.cc:492] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:30.998566 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:31.014815 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:31.047801 31378 tablet_bootstrap.cc:492] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:31.048280 31378 tablet_bootstrap.cc:492] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:10:31.048563 31378 ts_tablet_manager.cc:1367] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.001s
I20240501 20:10:31.049417 31378 raft_consensus.cc:348] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:31.049657 31378 raft_consensus.cc:729] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.053484 31378 consensus_queue.cc:260] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:31.057369 31378 ts_tablet_manager.cc:1398] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 20:10:31.058724 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:31.059581 24130 tablet_copy_service.cc:342] P 8559229e044b4e1ea661b881b3f9af50: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-d3c0aa45580749a4bd6fa7ccd635cb38 received from {username='slave'} at 127.13.251.129:53433
I20240501 20:10:31.059716 24130 tablet_copy_service.cc:433] P 8559229e044b4e1ea661b881b3f9af50: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-d3c0aa45580749a4bd6fa7ccd635cb38 on tablet d3c0aa45580749a4bd6fa7ccd635cb38 with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:31.121449 24108 raft_consensus.cc:1203] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:31.196734 20963 raft_consensus.cc:1203] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:31.239447 21358 raft_consensus.cc:1203] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:31.491816 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:31.513013 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:31.532582 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:31.546077 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:31.548218 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:31.550225 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:31.555783 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:31.580545 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:31.785516 20962 consensus_queue.cc:237] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:31.787894 21358 raft_consensus.cc:1261] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:31.788415 23975 raft_consensus.cc:1261] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:31.789276 31219 consensus_queue.cc:1035] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:31.789638 31219 consensus_queue.cc:1035] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:31.793416 31476 raft_consensus.cc:2932] T ca4fe4379c854ff1a0b8f9f17a6311cf P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:31.793972 23975 raft_consensus.cc:2932] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:31.797034 21358 raft_consensus.cc:2932] T ca4fe4379c854ff1a0b8f9f17a6311cf P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:31.800839 20823 catalog_manager.cc:5484] T ca4fe4379c854ff1a0b8f9f17a6311cf P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:31.809827 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4fe4379c854ff1a0b8f9f17a6311cf with delete_type TABLET_DATA_TOMBSTONED (TS 4af096f5a6a9440a94f6682b399efe20 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45322
I20240501 20:10:31.810433 31484 tablet_replica.cc:330] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:10:31.810706 31484 raft_consensus.cc:2229] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:31.810931 31484 raft_consensus.cc:2258] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:31.811612 31484 ts_tablet_manager.cc:1875] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:31.816550 31484 ts_tablet_manager.cc:1888] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:31.816761 31484 log.cc:1196] T ca4fe4379c854ff1a0b8f9f17a6311cf P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/ca4fe4379c854ff1a0b8f9f17a6311cf
I20240501 20:10:31.817521 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet ca4fe4379c854ff1a0b8f9f17a6311cf (table rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]) successfully deleted
I20240501 20:10:32.146229 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:32.156488 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:32.184077 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:32.185050 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:32.209591 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:32.230964 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:32.369925 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:32.406611 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:32.567495 23974 consensus_queue.cc:237] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:32.569612 23846 raft_consensus.cc:1261] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:32.569738 23711 raft_consensus.cc:1261] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:32.570183 31530 consensus_queue.cc:1035] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:32.570605 31397 consensus_queue.cc:1035] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:32.584864 31397 raft_consensus.cc:2932] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:32.588735 23846 raft_consensus.cc:2932] T 88f0ef64f05640c1a9e58e528149f8d0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:32.591784 23711 raft_consensus.cc:2932] T 88f0ef64f05640c1a9e58e528149f8d0 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:32.593894 20819 catalog_manager.cc:5484] T 88f0ef64f05640c1a9e58e528149f8d0 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:32.602727 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f0ef64f05640c1a9e58e528149f8d0 with delete_type TABLET_DATA_TOMBSTONED (TS 8559229e044b4e1ea661b881b3f9af50 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:32.617363 31582 tablet_replica.cc:330] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:32.618784 31582 raft_consensus.cc:2229] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:32.619102 31582 raft_consensus.cc:2258] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:32.619791 31582 ts_tablet_manager.cc:1875] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:32.629204 31582 ts_tablet_manager.cc:1888] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:32.629407 31582 log.cc:1196] T 88f0ef64f05640c1a9e58e528149f8d0 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/88f0ef64f05640c1a9e58e528149f8d0
I20240501 20:10:32.630156 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 88f0ef64f05640c1a9e58e528149f8d0 (table rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]) successfully deleted
I20240501 20:10:32.855778 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:32.869020 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:32.874796 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:32.894618 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:32.964787 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:32.972801 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:33.024839 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:33.100548 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:33.219166 21358 consensus_queue.cc:237] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:33.221472 24108 raft_consensus.cc:1261] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:33.222151 21228 raft_consensus.cc:1261] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:33.222574 31558 consensus_queue.cc:1035] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:33.222832 31646 consensus_queue.cc:1035] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:33.229445 31558 raft_consensus.cc:2932] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:33.232868 24108 raft_consensus.cc:2932] T 2b4384496d9048259bf353bd58a15008 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:33.241248 21228 raft_consensus.cc:2932] T 2b4384496d9048259bf353bd58a15008 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:33.242234 20818 catalog_manager.cc:5484] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:33.254405 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4384496d9048259bf353bd58a15008 with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51330
I20240501 20:10:33.255870 31674 tablet_replica.cc:330] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:10:33.256136 31674 raft_consensus.cc:2229] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:33.256429 31674 raft_consensus.cc:2258] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:33.257095 31674 ts_tablet_manager.cc:1875] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:33.261790 31674 ts_tablet_manager.cc:1888] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:33.261893 31674 log.cc:1196] T 2b4384496d9048259bf353bd58a15008 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b4384496d9048259bf353bd58a15008
I20240501 20:10:33.262626 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 2b4384496d9048259bf353bd58a15008 (table rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]) successfully deleted
I20240501 20:10:33.587752 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:33.602962 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:33.632732 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:33.659430 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:33.680497 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:33.702657 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:33.750106 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:33.840651 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:33.925484 21095 consensus_queue.cc:237] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:33.927863 23975 raft_consensus.cc:1261] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:33.928040 21225 raft_consensus.cc:1261] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:33.928303 30936 consensus_queue.cc:1035] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:33.928772 30936 consensus_queue.cc:1035] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:33.930894 31390 raft_consensus.cc:2932] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:33.931293 23975 raft_consensus.cc:2932] T 3a7e956aedc44f56b3bf629c80ff65e4 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:33.937587 20821 catalog_manager.cc:5484] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:33.961588 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a7e956aedc44f56b3bf629c80ff65e4 with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42940
I20240501 20:10:33.961375 21228 raft_consensus.cc:2932] T 3a7e956aedc44f56b3bf629c80ff65e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:33.963768 31759 tablet_replica.cc:330] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:10:33.964802 31759 raft_consensus.cc:2229] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:33.965073 31759 raft_consensus.cc:2258] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:33.965729 31759 ts_tablet_manager.cc:1875] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:33.990253 31759 ts_tablet_manager.cc:1888] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:33.990470 31759 log.cc:1196] T 3a7e956aedc44f56b3bf629c80ff65e4 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/3a7e956aedc44f56b3bf629c80ff65e4
I20240501 20:10:33.991263 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 3a7e956aedc44f56b3bf629c80ff65e4 (table rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]) successfully deleted
I20240501 20:10:34.314249 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:34.356377 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:34.373111 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:34.380959 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:34.459383 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:34.463465 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:34.476927 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:34.582206 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:34.691362 23846 consensus_queue.cc:237] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:34.694311 21225 raft_consensus.cc:1261] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:34.694847 20962 raft_consensus.cc:1261] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:34.694912 31396 consensus_queue.cc:1035] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:34.696038 31396 consensus_queue.cc:1035] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:34.699440 31837 raft_consensus.cc:2932] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:34.699849 21225 raft_consensus.cc:2932] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:34.699988 20962 raft_consensus.cc:2932] T 0722c64801734730abefe2a742da549e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:34.706243 20818 catalog_manager.cc:5484] T 0722c64801734730abefe2a742da549e P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:34.713135 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 0722c64801734730abefe2a742da549e with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:34.713892 31846 tablet_replica.cc:330] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:34.714159 31846 raft_consensus.cc:2229] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:34.714402 31846 raft_consensus.cc:2258] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:34.715276 31846 ts_tablet_manager.cc:1875] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:34.723191 31846 ts_tablet_manager.cc:1888] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:34.723421 31846 log.cc:1196] T 0722c64801734730abefe2a742da549e P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/0722c64801734730abefe2a742da549e
I20240501 20:10:34.724295 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 0722c64801734730abefe2a742da549e (table rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]) successfully deleted
I20240501 20:10:34.987836 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:34.991792 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:35.031738 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:35.044574 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:35.055280 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:35.069029 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:35.104087 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:35.122537 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:10:35.146696 21294 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 54081.3us
W20240501 20:10:35.231002 21032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 44509.2us
W20240501 20:10:35.247989 21033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 46148.2us
I20240501 20:10:35.356578 21095 consensus_queue.cc:237] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:35.359218 21358 raft_consensus.cc:1261] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:35.359429 24109 raft_consensus.cc:1261] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:35.360020 31910 consensus_queue.cc:1035] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:35.360425 31909 consensus_queue.cc:1035] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:35.378966 31910 raft_consensus.cc:2932] T 9d308589b223471bb2f20ff3c6a7f13e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:35.380548 21358 raft_consensus.cc:2932] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:35.382267 24108 raft_consensus.cc:2932] T 9d308589b223471bb2f20ff3c6a7f13e P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:35.389727 20818 catalog_manager.cc:5484] T 9d308589b223471bb2f20ff3c6a7f13e P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 2 to 3, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:35.399623 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d308589b223471bb2f20ff3c6a7f13e with delete_type TABLET_DATA_TOMBSTONED (TS 4af096f5a6a9440a94f6682b399efe20 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45322
I20240501 20:10:35.400269 31933 tablet_replica.cc:330] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:10:35.400521 31933 raft_consensus.cc:2229] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:35.400764 31933 raft_consensus.cc:2258] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:35.402487 31933 ts_tablet_manager.cc:1875] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:35.409492 31933 ts_tablet_manager.cc:1888] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:35.409613 31933 log.cc:1196] T 9d308589b223471bb2f20ff3c6a7f13e P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/9d308589b223471bb2f20ff3c6a7f13e
I20240501 20:10:35.411507 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 9d308589b223471bb2f20ff3c6a7f13e (table rebalancer_extra_table_10 [id=e1eecbe79d2f4bb7a34fcc87a2eae29e]) successfully deleted
I20240501 20:10:35.810400 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:35.818887 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:35.870628 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:35.871237 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:35.872156 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:35.875011 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240501 20:10:35.894779 23647 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 48617.3us
I20240501 20:10:35.920543 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:35.972786 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:36.166477 21097 consensus_queue.cc:237] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:36.168426 23975 raft_consensus.cc:1261] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.169006 31910 consensus_queue.cc:1035] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.169500 24108 raft_consensus.cc:1261] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.170078 31910 consensus_queue.cc:1035] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.172595 31910 raft_consensus.cc:2932] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:36.176517 24108 raft_consensus.cc:2932] T 20749a5ab2744578afd2978320e7fbf3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:36.176534 23975 raft_consensus.cc:2932] T 20749a5ab2744578afd2978320e7fbf3 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:36.182516 20826 catalog_manager.cc:5484] T 20749a5ab2744578afd2978320e7fbf3 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:36.186780 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 20749a5ab2744578afd2978320e7fbf3 with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:36.187340 32020 tablet_replica.cc:330] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:36.187625 32020 raft_consensus.cc:2229] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:36.188016 32020 raft_consensus.cc:2258] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:36.189190 32020 ts_tablet_manager.cc:1875] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:36.194607 32020 ts_tablet_manager.cc:1888] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:36.194859 32020 log.cc:1196] T 20749a5ab2744578afd2978320e7fbf3 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/20749a5ab2744578afd2978320e7fbf3
I20240501 20:10:36.196146 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 20749a5ab2744578afd2978320e7fbf3 (table rebalancer_extra_table_36 [id=5ecc8d0abc46475c8aad6578c8a56d99]) successfully deleted
I20240501 20:10:36.455893 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:36.478876 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:36.479367 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:36.481264 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:36.523500 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:36.589426 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:36.607707 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:36.711413 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:36.822319 21356 consensus_queue.cc:237] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:36.824435 20965 raft_consensus.cc:1261] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.824541 23974 raft_consensus.cc:1261] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.825120 31976 consensus_queue.cc:1035] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.826869 31976 consensus_queue.cc:1035] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.828076 32083 raft_consensus.cc:2932] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:36.829203 23975 raft_consensus.cc:2932] T 25c9aac0679a41579f390c49f546f66b P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:36.836071 20823 catalog_manager.cc:5484] T 25c9aac0679a41579f390c49f546f66b P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:36.838491 20965 raft_consensus.cc:2932] T 25c9aac0679a41579f390c49f546f66b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:36.852522 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c9aac0679a41579f390c49f546f66b with delete_type TABLET_DATA_TOMBSTONED (TS 8559229e044b4e1ea661b881b3f9af50 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:36.864531 32109 tablet_replica.cc:330] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:36.864822 32109 raft_consensus.cc:2229] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:36.865548 32109 raft_consensus.cc:2258] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:36.869864 32109 ts_tablet_manager.cc:1875] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:36.875221 32109 ts_tablet_manager.cc:1888] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:36.875322 32109 log.cc:1196] T 25c9aac0679a41579f390c49f546f66b P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/25c9aac0679a41579f390c49f546f66b
I20240501 20:10:36.876228 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 25c9aac0679a41579f390c49f546f66b (table rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]) successfully deleted
I20240501 20:10:37.213831 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:37.219089 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:37.236753 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:37.250598 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:37.299824 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:37.314440 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:37.332151 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:37.362059 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:37.454813 23846 consensus_queue.cc:237] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:37.456851 23975 raft_consensus.cc:1261] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.456949 21095 raft_consensus.cc:1261] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.457444 31948 consensus_queue.cc:1035] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.457751 31837 consensus_queue.cc:1035] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.460681 31948 raft_consensus.cc:2932] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:37.461335 23975 raft_consensus.cc:2932] T 6c31dffb0482438ebbc6243ad54752e9 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:37.467186 21095 raft_consensus.cc:2932] T 6c31dffb0482438ebbc6243ad54752e9 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:37.467763 20823 catalog_manager.cc:5484] T 6c31dffb0482438ebbc6243ad54752e9 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:37.491982 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c31dffb0482438ebbc6243ad54752e9 with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:37.493955 32199 tablet_replica.cc:330] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:37.494225 32199 raft_consensus.cc:2229] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:37.494506 32199 raft_consensus.cc:2258] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:37.495165 32199 ts_tablet_manager.cc:1875] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:37.501854 32199 ts_tablet_manager.cc:1888] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:37.502039 32199 log.cc:1196] T 6c31dffb0482438ebbc6243ad54752e9 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/6c31dffb0482438ebbc6243ad54752e9
I20240501 20:10:37.503085 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 6c31dffb0482438ebbc6243ad54752e9 (table rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]) successfully deleted
I20240501 20:10:38.122789 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:38.161835 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:38.189594 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:38.215004 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:38.270929 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:38.279692 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:38.332895 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:38.352165 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:38.448001 21225 consensus_queue.cc:237] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:38.450495 23846 raft_consensus.cc:1261] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:38.450495 21095 raft_consensus.cc:1261] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:38.450978 32273 consensus_queue.cc:1035] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:38.451239 31728 consensus_queue.cc:1035] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:38.456374 31728 raft_consensus.cc:2932] T 021fed14d5a140f3b42d359bae264f69 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:38.457381 23846 raft_consensus.cc:2932] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:38.468148 21095 raft_consensus.cc:2932] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:38.468258 20818 catalog_manager.cc:5484] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:38.518954 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 021fed14d5a140f3b42d359bae264f69 with delete_type TABLET_DATA_TOMBSTONED (TS cda19d6a81c0473db302535ce2ada7f0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54238
I20240501 20:10:38.523412 32303 tablet_replica.cc:330] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:10:38.528151 32303 raft_consensus.cc:2229] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:38.528573 32303 raft_consensus.cc:2258] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:38.530596 32303 ts_tablet_manager.cc:1875] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:38.566416 32303 ts_tablet_manager.cc:1888] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:38.566614 32303 log.cc:1196] T 021fed14d5a140f3b42d359bae264f69 P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/021fed14d5a140f3b42d359bae264f69
I20240501 20:10:38.567468 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 021fed14d5a140f3b42d359bae264f69 (table rebalance_test_table_3 [id=4049a891524a4edf9ddc53db7cfd28f6]) successfully deleted
I20240501 20:10:38.824515 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:38.840153 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:38.866221 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:38.910089 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:38.974187 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:39.005736 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:39.010519 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:39.060446 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:39.212018 21225 consensus_queue.cc:237] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:39.214143 20963 raft_consensus.cc:1261] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:39.214177 23846 raft_consensus.cc:1261] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:39.214720 32110 consensus_queue.cc:1035] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:39.215029 32293 consensus_queue.cc:1035] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:39.218092 32385 raft_consensus.cc:2932] T b472171e77e144ffa8fa4cdbd40a03a0 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:39.218684 23846 raft_consensus.cc:2932] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:39.219794 20963 raft_consensus.cc:2932] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:39.225457 20823 catalog_manager.cc:5484] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:39.236923 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet b472171e77e144ffa8fa4cdbd40a03a0 with delete_type TABLET_DATA_TOMBSTONED (TS 8559229e044b4e1ea661b881b3f9af50 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:39.237625 32395 tablet_replica.cc:330] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:39.237936 32395 raft_consensus.cc:2229] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:39.238221 32395 raft_consensus.cc:2258] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:39.238942 32395 ts_tablet_manager.cc:1875] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:39.245549 32395 ts_tablet_manager.cc:1888] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:39.245688 32395 log.cc:1196] T b472171e77e144ffa8fa4cdbd40a03a0 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/b472171e77e144ffa8fa4cdbd40a03a0
I20240501 20:10:39.246696 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet b472171e77e144ffa8fa4cdbd40a03a0 (table rebalancer_extra_table_22 [id=f8c55de938104e9c9885d55e421e1d0f]) successfully deleted
I20240501 20:10:39.511346 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:39.567708 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:39.654588 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:39.710407 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:39.726174 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:39.745875 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:39.777081 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:39.856228 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:40.020834 24109 consensus_queue.cc:237] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:40.032790 23846 raft_consensus.cc:1261] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:40.033130 23974 raft_consensus.cc:1261] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:40.033298 32455 consensus_queue.cc:1035] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:40.033915 32455 consensus_queue.cc:1035] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:40.036630 32455 raft_consensus.cc:2932] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:40.037173 23846 raft_consensus.cc:2932] T 8978a4c14f0a4446a979d3fc05248085 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:40.042973 20823 catalog_manager.cc:5484] T 8978a4c14f0a4446a979d3fc05248085 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 2 to 4, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New cstate: current_term: 2 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:40.044836 23975 raft_consensus.cc:2932] T 8978a4c14f0a4446a979d3fc05248085 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:40.073947 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet 8978a4c14f0a4446a979d3fc05248085 with delete_type TABLET_DATA_TOMBSTONED (TS 4af096f5a6a9440a94f6682b399efe20 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45322
I20240501 20:10:40.074793 32490 tablet_replica.cc:330] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: stopping tablet replica
I20240501 20:10:40.077107 32490 raft_consensus.cc:2229] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:40.077392 32490 raft_consensus.cc:2258] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:40.078334 32490 ts_tablet_manager.cc:1875] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:40.088032 32490 ts_tablet_manager.cc:1888] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 20:10:40.088212 32490 log.cc:1196] T 8978a4c14f0a4446a979d3fc05248085 P 4af096f5a6a9440a94f6682b399efe20: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-2/wal/wals/8978a4c14f0a4446a979d3fc05248085
I20240501 20:10:40.088977 20813 catalog_manager.cc:4830] TS 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): tablet 8978a4c14f0a4446a979d3fc05248085 (table rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]) successfully deleted
I20240501 20:10:40.123258 23975 consensus_queue.cc:237] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:40.126101 23711 raft_consensus.cc:1261] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:40.126137 23846 raft_consensus.cc:1261] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:40.126799 32084 consensus_queue.cc:1035] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:40.127242 23909 consensus_peers.cc:485] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f1a90f2878c433381aed777ac33551d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.127388 32412 consensus_queue.cc:1035] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:40.132539 32412 raft_consensus.cc:2932] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:40.133172 23846 raft_consensus.cc:2932] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:40.137301 23711 raft_consensus.cc:2932] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:40.139225 20823 catalog_manager.cc:5484] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:40.177488 21095 consensus_queue.cc:237] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:40.180213 20963 raft_consensus.cc:1261] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:40.180253 21031 consensus_peers.cc:485] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Couldn't send request to peer 744c5a532e724391830c0785d32eb2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5da8abdb3014753ac3cbef1d622069e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.180761 21355 raft_consensus.cc:1261] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:40.180752 32525 consensus_queue.cc:1035] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:40.181322 32525 consensus_queue.cc:1035] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:40.184631 32525 raft_consensus.cc:2932] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:40.185276 21355 raft_consensus.cc:2932] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:40.189910 20963 raft_consensus.cc:2932] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:40.191901 20817 catalog_manager.cc:5484] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:40.242828 21355 consensus_queue.cc:237] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:40.245738 23846 raft_consensus.cc:1261] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:40.245816 21095 raft_consensus.cc:1261] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:40.246308 21293 consensus_peers.cc:485] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 590a21c93ae543c8babdf9b89d3192e7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.246404 32083 consensus_queue.cc:1035] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:40.246670 32175 consensus_queue.cc:1035] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:40.249979 32175 raft_consensus.cc:2932] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:40.252215 21095 raft_consensus.cc:2932] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:40.257170 20823 catalog_manager.cc:5484] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:40.259980 23846 raft_consensus.cc:2932] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:40.327183 23975 consensus_queue.cc:237] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
W20240501 20:10:40.329949 23909 consensus_peers.cc:485] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef662013a0e4494dbe319efc1e99b03b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.330034 20963 raft_consensus.cc:1261] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:40.330111 23846 raft_consensus.cc:1261] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:40.330674 32412 consensus_queue.cc:1035] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:40.331238 32084 consensus_queue.cc:1035] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:40.334092 32084 raft_consensus.cc:2932] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:40.336100 20963 raft_consensus.cc:2932] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:40.340263 23846 raft_consensus.cc:2932] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:40.345808 20820 catalog_manager.cc:5484] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:40.386268 24109 consensus_queue.cc:237] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:40.388556 23846 raft_consensus.cc:1261] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:40.389129 23975 raft_consensus.cc:1261] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:40.389123 32456 consensus_queue.cc:1035] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:40.389703 32455 consensus_queue.cc:1035] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:40.389979 24044 consensus_peers.cc:485] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30ea740ef5534b089c77063a8d8b9aaa. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.394742 32456 raft_consensus.cc:2932] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:40.395733 23846 raft_consensus.cc:2932] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:40.400908 23975 raft_consensus.cc:2932] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:40.402701 20820 catalog_manager.cc:5484] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:40.441226 24109 consensus_queue.cc:237] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:40.443832 23975 raft_consensus.cc:1261] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:40.443833 20963 raft_consensus.cc:1261] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:40.444154 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f3951bb18524e41aa6836122e97e7f5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.444469 31537 consensus_queue.cc:1035] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:40.444721 32456 consensus_queue.cc:1035] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:40.448038 31537 raft_consensus.cc:2932] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:40.448836 20963 raft_consensus.cc:2932] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:40.454241 20817 catalog_manager.cc:5484] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:40.457175 23975 raft_consensus.cc:2932] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:40.616727 32622 ts_tablet_manager.cc:902] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:40.618227 32622 tablet_copy_client.cc:308] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:10:40.619056 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet 0f1a90f2878c433381aed777ac33551d from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:58437)
I20240501 20:10:40.619258 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet 0f1a90f2878c433381aed777ac33551d from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:58437: session id = feac73db0a3b4aafa374ca5dfd322b08-0f1a90f2878c433381aed777ac33551d
I20240501 20:10:40.623445 23997 tablet_copy_source_session.cc:205] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:40.625563 32622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1a90f2878c433381aed777ac33551d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:40.631983 32622 tablet_copy_client.cc:783] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:40.632369 32622 tablet_copy_client.cc:647] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:40.635054 32622 tablet_copy_client.cc:523] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:40.640420 32622 tablet_bootstrap.cc:492] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:40.647419 32622 tablet_bootstrap.cc:492] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:40.647974 32622 tablet_bootstrap.cc:492] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:10:40.648275 32622 ts_tablet_manager.cc:1367] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:10:40.649171 32622 raft_consensus.cc:348] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:40.649394 32622 raft_consensus.cc:729] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:40.649636 32622 consensus_queue.cc:260] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:40.650274 32622 ts_tablet_manager.cc:1398] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:40.650892 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-0f1a90f2878c433381aed777ac33551d received from {username='slave'} at 127.13.251.130:58437
I20240501 20:10:40.651023 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-0f1a90f2878c433381aed777ac33551d on tablet 0f1a90f2878c433381aed777ac33551d with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:10:40.663579 32640 ts_tablet_manager.cc:902] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:10:40.677054 32640 tablet_copy_client.cc:308] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:10:40.678035 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet e5da8abdb3014753ac3cbef1d622069e from peer 744c5a532e724391830c0785d32eb2fb ({username='slave'} at 127.13.251.135:53413)
I20240501 20:10:40.678193 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet e5da8abdb3014753ac3cbef1d622069e from peer 744c5a532e724391830c0785d32eb2fb at {username='slave'} at 127.13.251.135:53413: session id = 744c5a532e724391830c0785d32eb2fb-e5da8abdb3014753ac3cbef1d622069e
I20240501 20:10:40.680883 21118 tablet_copy_source_session.cc:205] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:40.682947 32640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5da8abdb3014753ac3cbef1d622069e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:40.689534 32640 tablet_copy_client.cc:783] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:40.689774 32640 tablet_copy_client.cc:647] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:40.691612 32640 tablet_copy_client.cc:523] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:40.695398 32640 tablet_bootstrap.cc:492] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:40.751727 32640 tablet_bootstrap.cc:492] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:40.752202 32640 tablet_bootstrap.cc:492] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: Bootstrap complete.
I20240501 20:10:40.752509 32640 ts_tablet_manager.cc:1367] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.057s user 0.042s sys 0.005s
I20240501 20:10:40.753317 32640 raft_consensus.cc:348] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:40.753536 32640 raft_consensus.cc:729] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:40.753729 32640 consensus_queue.cc:260] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:40.754281 32640 ts_tablet_manager.cc:1398] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:40.755334 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session 744c5a532e724391830c0785d32eb2fb-e5da8abdb3014753ac3cbef1d622069e received from {username='slave'} at 127.13.251.135:53413
I20240501 20:10:40.755443 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session 744c5a532e724391830c0785d32eb2fb-e5da8abdb3014753ac3cbef1d622069e on tablet e5da8abdb3014753ac3cbef1d622069e with peer 744c5a532e724391830c0785d32eb2fb
I20240501 20:10:40.794101 32662 ts_tablet_manager.cc:902] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:40.802161 32662 tablet_copy_client.cc:308] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:10:40.803051 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 590a21c93ae543c8babdf9b89d3192e7 from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:37097)
I20240501 20:10:40.803197 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 590a21c93ae543c8babdf9b89d3192e7 from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:37097: session id = 51bf640e895b4e5eb91bdf7509ea7e46-590a21c93ae543c8babdf9b89d3192e7
I20240501 20:10:40.805636 21378 tablet_copy_source_session.cc:205] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:40.811937 32662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590a21c93ae543c8babdf9b89d3192e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:40.823328 32662 tablet_copy_client.cc:783] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:40.823601 32662 tablet_copy_client.cc:647] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:40.829929 32671 ts_tablet_manager.cc:902] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:40.832485 32662 tablet_copy_client.cc:523] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:40.837042 32662 tablet_bootstrap.cc:492] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:40.846583 32671 tablet_copy_client.cc:308] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.136:44267
I20240501 20:10:40.847435 24130 tablet_copy_service.cc:140] P 8559229e044b4e1ea661b881b3f9af50: Received BeginTabletCopySession request for tablet 3f3951bb18524e41aa6836122e97e7f5 from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:53727)
I20240501 20:10:40.847604 24130 tablet_copy_service.cc:161] P 8559229e044b4e1ea661b881b3f9af50: Beginning new tablet copy session on tablet 3f3951bb18524e41aa6836122e97e7f5 from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:53727: session id = 4af096f5a6a9440a94f6682b399efe20-3f3951bb18524e41aa6836122e97e7f5
I20240501 20:10:40.849867 24130 tablet_copy_source_session.cc:205] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:40.850966 32671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3951bb18524e41aa6836122e97e7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:40.852104 32662 tablet_bootstrap.cc:492] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:40.852555 32662 tablet_bootstrap.cc:492] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:10:40.852835 32662 ts_tablet_manager.cc:1367] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.006s
I20240501 20:10:40.853854 32662 raft_consensus.cc:348] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:40.854123 32662 raft_consensus.cc:729] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:40.856973 32672 ts_tablet_manager.cc:902] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:40.857124 32671 tablet_copy_client.cc:783] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:40.857352 32671 tablet_copy_client.cc:647] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:40.854451 32662 consensus_queue.cc:260] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:40.858652 32662 ts_tablet_manager.cc:1398] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:10:40.859716 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-590a21c93ae543c8babdf9b89d3192e7 received from {username='slave'} at 127.13.251.133:37097
I20240501 20:10:40.859820 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-590a21c93ae543c8babdf9b89d3192e7 on tablet 590a21c93ae543c8babdf9b89d3192e7 with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:10:40.882179 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:40.890300 32672 tablet_copy_client.cc:308] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.136:44267
I20240501 20:10:40.890944 32671 tablet_copy_client.cc:523] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:40.892088 24130 tablet_copy_service.cc:140] P 8559229e044b4e1ea661b881b3f9af50: Received BeginTabletCopySession request for tablet 30ea740ef5534b089c77063a8d8b9aaa from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:52271)
I20240501 20:10:40.892273 24130 tablet_copy_service.cc:161] P 8559229e044b4e1ea661b881b3f9af50: Beginning new tablet copy session on tablet 30ea740ef5534b089c77063a8d8b9aaa from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:52271: session id = 61380023001a4ee9ada9ec2c6c1b7159-30ea740ef5534b089c77063a8d8b9aaa
I20240501 20:10:40.900386 24130 tablet_copy_source_session.cc:205] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:40.902715 32672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ea740ef5534b089c77063a8d8b9aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:40.899562 32671 tablet_bootstrap.cc:492] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:40.915098 32672 tablet_copy_client.cc:783] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:40.916204 32672 tablet_copy_client.cc:647] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:40.917897 32672 tablet_copy_client.cc:523] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:40.919163 32622 ts_tablet_manager.cc:902] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:40.926574 32672 tablet_bootstrap.cc:492] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:10:40.928768 32671 tablet_bootstrap.cc:492] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:40.929123 32671 tablet_bootstrap.cc:492] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:10:40.929356 32671 ts_tablet_manager.cc:1367] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20240501 20:10:40.933466 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:40.935693 32671 raft_consensus.cc:348] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:40.935917 32671 raft_consensus.cc:729] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:40.937069 32671 consensus_queue.cc:260] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:40.938138 32671 ts_tablet_manager.cc:1398] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:10:40.943499 24130 tablet_copy_service.cc:342] P 8559229e044b4e1ea661b881b3f9af50: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-3f3951bb18524e41aa6836122e97e7f5 received from {username='slave'} at 127.13.251.131:53727
I20240501 20:10:40.943645 24130 tablet_copy_service.cc:433] P 8559229e044b4e1ea661b881b3f9af50: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-3f3951bb18524e41aa6836122e97e7f5 on tablet 3f3951bb18524e41aa6836122e97e7f5 with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:10:40.945905 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:40.967106 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:40.970203 32672 tablet_bootstrap.cc:492] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:40.973656 32672 tablet_bootstrap.cc:492] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:10:40.973891 32672 ts_tablet_manager.cc:1367] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.008s
I20240501 20:10:40.974828 32672 raft_consensus.cc:348] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:40.975083 32672 raft_consensus.cc:729] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:10:40.975405 32672 consensus_queue.cc:260] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:40.982568 32672 ts_tablet_manager.cc:1398] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:10:40.986881 24130 tablet_copy_service.cc:342] P 8559229e044b4e1ea661b881b3f9af50: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-30ea740ef5534b089c77063a8d8b9aaa received from {username='slave'} at 127.13.251.132:52271
I20240501 20:10:40.987016 24130 tablet_copy_service.cc:433] P 8559229e044b4e1ea661b881b3f9af50: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-30ea740ef5534b089c77063a8d8b9aaa on tablet 30ea740ef5534b089c77063a8d8b9aaa with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:10:40.996353 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:41.027767 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:41.057731 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:41.058544 21097 raft_consensus.cc:1203] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:41.105430 23975 raft_consensus.cc:1203] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:41.138157 32622 tablet_copy_client.cc:308] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:10:41.138794 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet ef662013a0e4494dbe319efc1e99b03b from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:58437)
I20240501 20:10:41.138945 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet ef662013a0e4494dbe319efc1e99b03b from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:58437: session id = feac73db0a3b4aafa374ca5dfd322b08-ef662013a0e4494dbe319efc1e99b03b
I20240501 20:10:41.141851 23997 tablet_copy_source_session.cc:205] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:41.143182 32622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef662013a0e4494dbe319efc1e99b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:41.148594 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:41.161612 32622 tablet_copy_client.cc:783] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:41.161891 32622 tablet_copy_client.cc:647] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:41.174137 32622 tablet_copy_client.cc:523] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:41.180050 32622 tablet_bootstrap.cc:492] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:41.222507 32622 tablet_bootstrap.cc:492] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:41.222893 32622 tablet_bootstrap.cc:492] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:10:41.223152 32622 ts_tablet_manager.cc:1367] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.002s
I20240501 20:10:41.223819 32622 raft_consensus.cc:348] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:41.224036 32622 raft_consensus.cc:729] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:41.225236 32622 consensus_queue.cc:260] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:41.226011 32622 ts_tablet_manager.cc:1398] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:41.231796 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-ef662013a0e4494dbe319efc1e99b03b received from {username='slave'} at 127.13.251.130:58437
I20240501 20:10:41.231928 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-ef662013a0e4494dbe319efc1e99b03b on tablet ef662013a0e4494dbe319efc1e99b03b with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:10:41.287379 23710 raft_consensus.cc:1203] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:41.323820 21095 raft_consensus.cc:1203] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:41.398541 21358 raft_consensus.cc:1203] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:41.443166 21228 raft_consensus.cc:1203] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:41.575301 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:41.605768 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:41.623517 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:41.653090 23957 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:41.678884 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:41.699452 24089 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:41.719695 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:41.779170 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:41.842222 24108 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30ea740ef5534b089c77063a8d8b9aaa"
dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38346
I20240501 20:10:41.842496 24108 raft_consensus.cc:595] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:42.005853 32456 raft_consensus.cc:979] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: : Instructing follower 744c5a532e724391830c0785d32eb2fb to start an election
I20240501 20:10:42.006031 32456 raft_consensus.cc:1067] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Signalling peer 744c5a532e724391830c0785d32eb2fb to start an election
I20240501 20:10:42.006970 23974 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30ea740ef5534b089c77063a8d8b9aaa"
dest_uuid: "744c5a532e724391830c0785d32eb2fb"
from {username='slave'} at 127.13.251.136:48117
I20240501 20:10:42.007263 23974 raft_consensus.cc:482] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:42.007378 23974 raft_consensus.cc:3037] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:42.010694 23974 raft_consensus.cc:504] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:42.012113 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ea740ef5534b089c77063a8d8b9aaa" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:10:42.012159 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ea740ef5534b089c77063a8d8b9aaa" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:10:42.012354 23843 raft_consensus.cc:3037] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:42.012348 24108 raft_consensus.cc:3032] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:42.012421 24108 raft_consensus.cc:729] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:10:42.012698 24108 consensus_queue.cc:260] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:42.012962 21355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ea740ef5534b089c77063a8d8b9aaa" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:10:42.013162 21355 raft_consensus.cc:3037] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:42.013197 24108 raft_consensus.cc:3037] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:42.015028 23843 raft_consensus.cc:2454] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 2.
I20240501 20:10:42.015686 21355 raft_consensus.cc:2454] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 2.
I20240501 20:10:42.017019 23912 leader_election.cc:302] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 744c5a532e724391830c0785d32eb2fb, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters:
I20240501 20:10:42.017266 32412 raft_consensus.cc:2781] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:42.017360 32412 raft_consensus.cc:686] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 2 LEADER]: Becoming Leader. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Running, Role: LEADER
I20240501 20:10:42.017596 32412 consensus_queue.cc:237] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:42.020814 24108 raft_consensus.cc:2454] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744c5a532e724391830c0785d32eb2fb in term 2.
I20240501 20:10:42.021672 23974 leader_election.cc:288] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 2 election: Requested vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:42.045781 20823 catalog_manager.cc:5484] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb reported cstate change: term changed from 1 to 2, leader changed from 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) to 744c5a532e724391830c0785d32eb2fb (127.13.251.135). New cstate: current_term: 2 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
W20240501 20:10:42.184906 21163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 82597.6us
I20240501 20:10:42.266851 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:42.294850 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:42.327865 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:42.328152 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:42.359894 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:42.398947 21358 raft_consensus.cc:1261] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:42.401275 32084 consensus_queue.cc:1035] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:42.444677 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:42.447750 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:42.464812 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:42.495903 24109 raft_consensus.cc:1261] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:42.496488 32084 consensus_queue.cc:1035] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:42.526085 23846 raft_consensus.cc:1261] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:42.526624 32412 consensus_queue.cc:1035] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:42.913293 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:42.936636 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:43.004325 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:43.115093 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:10:43.137132 21034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 160231us
I20240501 20:10:43.157377 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:43.183801 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:43.195013 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:43.237298 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:43.393671 21355 consensus_queue.cc:237] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:43.395785 23709 raft_consensus.cc:1261] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:43.395803 21097 raft_consensus.cc:1261] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:43.396497 472 consensus_queue.cc:1035] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:43.396951 32546 consensus_queue.cc:1035] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:43.400736 472 raft_consensus.cc:2932] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:43.405956 23709 raft_consensus.cc:2932] T 590a21c93ae543c8babdf9b89d3192e7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:43.409847 21095 raft_consensus.cc:2932] T 590a21c93ae543c8babdf9b89d3192e7 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:43.414463 20817 catalog_manager.cc:5484] T 590a21c93ae543c8babdf9b89d3192e7 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:43.429994 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 590a21c93ae543c8babdf9b89d3192e7 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:10:43.430668 504 tablet_replica.cc:330] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:10:43.430887 504 raft_consensus.cc:2229] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:43.431087 504 raft_consensus.cc:2258] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:43.431818 504 ts_tablet_manager.cc:1875] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:43.440788 504 ts_tablet_manager.cc:1888] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:43.440959 504 log.cc:1196] T 590a21c93ae543c8babdf9b89d3192e7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/590a21c93ae543c8babdf9b89d3192e7
I20240501 20:10:43.442054 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 590a21c93ae543c8babdf9b89d3192e7 (table rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]) successfully deleted
I20240501 20:10:43.743392 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:43.745949 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:43.756296 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:43.767997 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:43.807564 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:43.868849 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:43.904491 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:43.927001 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:44.087740 23975 consensus_queue.cc:237] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:44.089854 21095 raft_consensus.cc:1261] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:44.089996 23710 raft_consensus.cc:1261] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:44.090600 32412 consensus_queue.cc:1035] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:44.091310 32084 consensus_queue.cc:1035] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:44.094090 32084 raft_consensus.cc:2932] T 0f1a90f2878c433381aed777ac33551d P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:44.099205 23710 raft_consensus.cc:2932] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:44.108649 21097 raft_consensus.cc:2932] T 0f1a90f2878c433381aed777ac33551d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:44.109939 20826 catalog_manager.cc:5484] T 0f1a90f2878c433381aed777ac33551d P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:44.135862 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1a90f2878c433381aed777ac33551d with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:10:44.136515 596 tablet_replica.cc:330] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:10:44.136758 596 raft_consensus.cc:2229] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:44.136971 596 raft_consensus.cc:2258] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:44.137645 596 ts_tablet_manager.cc:1875] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:44.145119 596 ts_tablet_manager.cc:1888] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:44.145277 596 log.cc:1196] T 0f1a90f2878c433381aed777ac33551d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/0f1a90f2878c433381aed777ac33551d
I20240501 20:10:44.147817 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 0f1a90f2878c433381aed777ac33551d (table rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]) successfully deleted
I20240501 20:10:44.368836 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:44.379884 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:44.391057 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:44.400357 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:44.422144 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:44.468397 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:44.516201 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:44.550509 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:44.682700 24107 consensus_queue.cc:237] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:44.685048 21225 raft_consensus.cc:1261] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:44.685588 20965 raft_consensus.cc:1261] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:44.686331 352 consensus_queue.cc:1035] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:44.686909 352 consensus_queue.cc:1035] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:44.689396 352 raft_consensus.cc:2932] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:44.689954 21225 raft_consensus.cc:2932] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:44.690127 20965 raft_consensus.cc:2932] T d3c0aa45580749a4bd6fa7ccd635cb38 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:44.696424 20818 catalog_manager.cc:5484] T d3c0aa45580749a4bd6fa7ccd635cb38 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:44.713546 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet d3c0aa45580749a4bd6fa7ccd635cb38 with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:44.726775 678 tablet_replica.cc:330] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:44.727005 678 raft_consensus.cc:2229] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:44.727252 678 raft_consensus.cc:2258] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:44.728063 678 ts_tablet_manager.cc:1875] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:44.734161 678 ts_tablet_manager.cc:1888] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:44.734355 678 log.cc:1196] T d3c0aa45580749a4bd6fa7ccd635cb38 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/d3c0aa45580749a4bd6fa7ccd635cb38
I20240501 20:10:44.735083 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet d3c0aa45580749a4bd6fa7ccd635cb38 (table rebalancer_extra_table_24 [id=f61478b1bcc94db38f84d122c5896b7f]) successfully deleted
I20240501 20:10:45.139394 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:45.167797 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:45.171805 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:45.204260 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:45.238273 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:45.249943 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:45.327363 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240501 20:10:45.359109 24044 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 80534.5us
I20240501 20:10:45.361732 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:45.463450 21097 consensus_queue.cc:237] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:45.468794 20965 raft_consensus.cc:1261] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.468865 23974 raft_consensus.cc:1261] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.469328 462 consensus_queue.cc:1035] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.469678 32014 consensus_queue.cc:1035] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.474736 462 raft_consensus.cc:2932] T e5da8abdb3014753ac3cbef1d622069e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:45.475229 20965 raft_consensus.cc:2932] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:45.482764 20818 catalog_manager.cc:5484] T e5da8abdb3014753ac3cbef1d622069e P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:45.486124 23975 raft_consensus.cc:2932] T e5da8abdb3014753ac3cbef1d622069e P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:45.512301 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet e5da8abdb3014753ac3cbef1d622069e with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42940
I20240501 20:10:45.532912 763 tablet_replica.cc:330] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:10:45.533277 763 raft_consensus.cc:2229] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:45.533834 763 raft_consensus.cc:2258] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:45.534708 763 ts_tablet_manager.cc:1875] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:45.555763 763 ts_tablet_manager.cc:1888] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:45.555902 763 log.cc:1196] T e5da8abdb3014753ac3cbef1d622069e P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/e5da8abdb3014753ac3cbef1d622069e
I20240501 20:10:45.556628 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet e5da8abdb3014753ac3cbef1d622069e (table rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]) successfully deleted
I20240501 20:10:45.865151 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:45.870637 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:45.875304 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:45.924916 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:45.926393 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:45.957968 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:10:45.975242 23912 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 104240us
I20240501 20:10:46.000787 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:46.084825 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:46.196074 23974 consensus_queue.cc:237] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:46.198007 23846 raft_consensus.cc:1261] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:46.198216 21095 raft_consensus.cc:1261] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:46.198561 830 consensus_queue.cc:1035] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:46.198796 769 consensus_queue.cc:1035] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:46.201035 769 raft_consensus.cc:2932] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:46.201596 23846 raft_consensus.cc:2932] T ef662013a0e4494dbe319efc1e99b03b P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:46.201694 21095 raft_consensus.cc:2932] T ef662013a0e4494dbe319efc1e99b03b P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:46.207144 20825 catalog_manager.cc:5484] T ef662013a0e4494dbe319efc1e99b03b P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:46.227393 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet ef662013a0e4494dbe319efc1e99b03b with delete_type TABLET_DATA_TOMBSTONED (TS cda19d6a81c0473db302535ce2ada7f0 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54238
I20240501 20:10:46.243508 860 tablet_replica.cc:330] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:10:46.243996 860 raft_consensus.cc:2229] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:46.244226 860 raft_consensus.cc:2258] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:46.244818 860 ts_tablet_manager.cc:1875] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.249984 860 ts_tablet_manager.cc:1888] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:46.250114 860 log.cc:1196] T ef662013a0e4494dbe319efc1e99b03b P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef662013a0e4494dbe319efc1e99b03b
I20240501 20:10:46.250833 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet ef662013a0e4494dbe319efc1e99b03b (table rebalancer_extra_table_34 [id=c40f6fb69b2b4c2495e2a95d293abe73]) successfully deleted
I20240501 20:10:46.398365 20965 consensus_queue.cc:237] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
W20240501 20:10:46.400941 20902 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73fcc9ac412643088c996b76fbb1e680. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.401120 21228 raft_consensus.cc:1261] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.402350 406 consensus_queue.cc:1035] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:46.404225 24109 raft_consensus.cc:1261] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.405177 404 consensus_queue.cc:1035] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:46.407591 404 raft_consensus.cc:2932] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:46.408140 24109 raft_consensus.cc:2932] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:46.414615 21228 raft_consensus.cc:2932] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:46.417656 20819 catalog_manager.cc:5484] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:46.454351 23846 consensus_queue.cc:237] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:46.456856 21358 raft_consensus.cc:1261] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.457124 21095 raft_consensus.cc:1261] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.457875 31948 consensus_queue.cc:1035] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:46.458141 806 consensus_queue.cc:1035] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:46.461148 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7498fc5cf5144d58712eb2c8f188c5e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.469406 806 raft_consensus.cc:2932] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.469878 21095 raft_consensus.cc:2932] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.476006 21358 raft_consensus.cc:2932] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.478147 20819 catalog_manager.cc:5484] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.577932 21228 consensus_queue.cc:237] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:46.581144 21358 raft_consensus.cc:1261] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:46.582000 20965 raft_consensus.cc:1261] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:46.582863 685 consensus_queue.cc:1035] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:46.583914 962 consensus_queue.cc:1035] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:46.584383 21164 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 -> Peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Couldn't send request to peer c6dcaa8efff642dcb7c2c6e8be3a65f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10a4d24c5c844c8c9ccd191b1a2b0b00. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.599582 685 raft_consensus.cc:2932] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:46.606048 20824 catalog_manager.cc:5484] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:46.609704 21358 raft_consensus.cc:2932] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:46.604249 20962 raft_consensus.cc:2932] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:46.653160 23709 consensus_queue.cc:237] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:46.656445 23974 raft_consensus.cc:1261] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.656769 20962 raft_consensus.cc:1261] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.657526 807 consensus_queue.cc:1035] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:46.658735 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46677b6d96bf45d98d061e572a85ebdc. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.659030 807 consensus_queue.cc:1035] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:46.662233 982 raft_consensus.cc:2932] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.663224 23975 raft_consensus.cc:2932] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.667475 20962 raft_consensus.cc:2932] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.673518 20824 catalog_manager.cc:5484] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.709348 21228 consensus_queue.cc:237] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:46.712059 24109 raft_consensus.cc:1261] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.712659 21095 raft_consensus.cc:1261] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.712731 473 consensus_queue.cc:1035] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:46.713009 21163 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d75d2ef39a5347e2b6fbe3145c207280. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.714123 473 consensus_queue.cc:1035] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:46.720760 962 raft_consensus.cc:2932] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:46.721422 24109 raft_consensus.cc:2932] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:46.726701 20824 catalog_manager.cc:5484] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:46.727345 21095 raft_consensus.cc:2932] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:46.771508 21357 consensus_queue.cc:237] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:46.773948 24109 raft_consensus.cc:1261] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.774230 20962 raft_consensus.cc:1261] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:46.774289 21291 consensus_peers.cc:485] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Couldn't send request to peer 744c5a532e724391830c0785d32eb2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ecad173c71744eca8ca537ad152f95f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.774566 916 consensus_queue.cc:1035] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:46.775089 916 consensus_queue.cc:1035] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:46.779160 718 raft_consensus.cc:2932] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:46.779769 20962 raft_consensus.cc:2932] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:46.786746 20819 catalog_manager.cc:5484] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index -1 to 2, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:46.786825 24109 raft_consensus.cc:2932] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:46.825906 20962 consensus_queue.cc:237] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:46.828244 21095 raft_consensus.cc:1261] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:46.828399 21357 raft_consensus.cc:1261] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:46.828541 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c018d1b99ad749e59a2cf338001d38e4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.828913 923 consensus_queue.cc:1035] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:46.829177 404 consensus_queue.cc:1035] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:46.832926 404 raft_consensus.cc:2932] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.833694 21357 raft_consensus.cc:2932] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.838791 21095 raft_consensus.cc:2932] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.840895 20819 catalog_manager.cc:5484] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:46.871793 1053 ts_tablet_manager.cc:902] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:46.879880 1053 tablet_copy_client.cc:308] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:41259
I20240501 20:10:46.880570 20987 tablet_copy_service.cc:140] P cda19d6a81c0473db302535ce2ada7f0: Received BeginTabletCopySession request for tablet 73fcc9ac412643088c996b76fbb1e680 from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:53673)
I20240501 20:10:46.880725 20987 tablet_copy_service.cc:161] P cda19d6a81c0473db302535ce2ada7f0: Beginning new tablet copy session on tablet 73fcc9ac412643088c996b76fbb1e680 from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:53673: session id = 51bf640e895b4e5eb91bdf7509ea7e46-73fcc9ac412643088c996b76fbb1e680
I20240501 20:10:46.883214 20987 tablet_copy_source_session.cc:205] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:46.884452 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73fcc9ac412643088c996b76fbb1e680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:46.884622 21228 consensus_queue.cc:237] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:46.887187 21357 raft_consensus.cc:1261] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:46.887333 21164 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 -> Peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Couldn't send request to peer c6dcaa8efff642dcb7c2c6e8be3a65f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d75b25758954877ab36e8029fb29463. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.887712 473 consensus_queue.cc:1035] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:46.888096 21095 raft_consensus.cc:1261] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:46.888706 473 consensus_queue.cc:1035] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:46.892302 1053 tablet_copy_client.cc:783] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:46.892587 1053 tablet_copy_client.cc:647] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:46.897395 685 raft_consensus.cc:2932] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:46.898927 1065 ts_tablet_manager.cc:902] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:46.901811 21357 raft_consensus.cc:2932] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:46.903672 1053 tablet_copy_client.cc:523] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:46.906980 1053 tablet_bootstrap.cc:492] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:46.908927 21095 raft_consensus.cc:2932] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:46.909627 20824 catalog_manager.cc:5484] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:46.924476 1053 tablet_bootstrap.cc:492] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:46.924868 1053 tablet_bootstrap.cc:492] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:10:46.925086 1053 ts_tablet_manager.cc:1367] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240501 20:10:46.925789 1053 raft_consensus.cc:348] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:46.925987 1053 raft_consensus.cc:729] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:46.926190 1053 consensus_queue.cc:260] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:46.926821 1053 ts_tablet_manager.cc:1398] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:46.927711 20987 tablet_copy_service.cc:342] P cda19d6a81c0473db302535ce2ada7f0: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-73fcc9ac412643088c996b76fbb1e680 received from {username='slave'} at 127.13.251.133:53673
I20240501 20:10:46.927812 20987 tablet_copy_service.cc:433] P cda19d6a81c0473db302535ce2ada7f0: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-73fcc9ac412643088c996b76fbb1e680 on tablet 73fcc9ac412643088c996b76fbb1e680 with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:10:46.937042 1065 tablet_copy_client.cc:308] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:10:46.937925 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet a7498fc5cf5144d58712eb2c8f188c5e from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:54857)
I20240501 20:10:46.938087 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet a7498fc5cf5144d58712eb2c8f188c5e from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:54857: session id = 4af096f5a6a9440a94f6682b399efe20-a7498fc5cf5144d58712eb2c8f188c5e
I20240501 20:10:46.940178 23866 tablet_copy_source_session.cc:205] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:46.942932 1065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7498fc5cf5144d58712eb2c8f188c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:46.949988 1065 tablet_copy_client.cc:783] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:46.950239 1065 tablet_copy_client.cc:647] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:46.951987 1065 tablet_copy_client.cc:523] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:46.955395 1065 tablet_bootstrap.cc:492] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:46.975159 1065 tablet_bootstrap.cc:492] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:46.975857 1065 tablet_bootstrap.cc:492] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:10:46.976215 1065 ts_tablet_manager.cc:1367] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.009s
I20240501 20:10:46.977192 1065 raft_consensus.cc:348] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:46.980584 1065 raft_consensus.cc:729] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:46.980940 1065 consensus_queue.cc:260] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:46.981711 1065 ts_tablet_manager.cc:1398] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:10:46.982522 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-a7498fc5cf5144d58712eb2c8f188c5e received from {username='slave'} at 127.13.251.131:54857
I20240501 20:10:46.982635 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-a7498fc5cf5144d58712eb2c8f188c5e on tablet a7498fc5cf5144d58712eb2c8f188c5e with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:10:47.001322 21357 consensus_queue.cc:237] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:47.003801 21095 raft_consensus.cc:1261] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:47.004082 21293 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 748b4fd500f84a359bd8ee05b861300d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.004372 21228 raft_consensus.cc:1261] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.004455 1022 consensus_queue.cc:1035] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.004827 1023 consensus_queue.cc:1035] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.016916 1022 raft_consensus.cc:2932] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:47.021415 21095 raft_consensus.cc:2932] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:47.023829 21228 raft_consensus.cc:2932] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:47.024214 20824 catalog_manager.cc:5484] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:47.090461 23846 consensus_queue.cc:237] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:47.093691 23710 raft_consensus.cc:1261] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:47.094226 945 consensus_queue.cc:1035] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:47.094519 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b57a80759944301a033140e482aec36. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.096148 24109 raft_consensus.cc:1261] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:47.096982 806 consensus_queue.cc:1035] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:47.108933 806 raft_consensus.cc:2932] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:47.114663 24107 raft_consensus.cc:2932] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:47.116824 1110 ts_tablet_manager.cc:902] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:47.123056 20817 catalog_manager.cc:5484] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:47.123669 23710 raft_consensus.cc:2932] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:47.135938 1110 tablet_copy_client.cc:308] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:10:47.136626 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet 10a4d24c5c844c8c9ccd191b1a2b0b00 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 ({username='slave'} at 127.13.251.134:60949)
I20240501 20:10:47.136782 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet 10a4d24c5c844c8c9ccd191b1a2b0b00 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 at {username='slave'} at 127.13.251.134:60949: session id = c6dcaa8efff642dcb7c2c6e8be3a65f7-10a4d24c5c844c8c9ccd191b1a2b0b00
I20240501 20:10:47.140200 21248 tablet_copy_source_session.cc:205] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.141539 1110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a4d24c5c844c8c9ccd191b1a2b0b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.147022 1110 tablet_copy_client.cc:783] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.147320 1110 tablet_copy_client.cc:647] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.150427 1110 tablet_copy_client.cc:523] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.153192 1110 tablet_bootstrap.cc:492] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:47.155716 1065 ts_tablet_manager.cc:902] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:10:47.159960 1065 tablet_copy_client.cc:308] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.133:37155
I20240501 20:10:47.160692 23734 tablet_copy_service.cc:140] P 51bf640e895b4e5eb91bdf7509ea7e46: Received BeginTabletCopySession request for tablet 46677b6d96bf45d98d061e572a85ebdc from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:50321)
I20240501 20:10:47.160851 23734 tablet_copy_service.cc:161] P 51bf640e895b4e5eb91bdf7509ea7e46: Beginning new tablet copy session on tablet 46677b6d96bf45d98d061e572a85ebdc from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:50321: session id = 4af096f5a6a9440a94f6682b399efe20-46677b6d96bf45d98d061e572a85ebdc
I20240501 20:10:47.163200 23734 tablet_copy_source_session.cc:205] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.167920 1065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46677b6d96bf45d98d061e572a85ebdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.179643 1065 tablet_copy_client.cc:783] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.179890 1065 tablet_copy_client.cc:647] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.185442 1065 tablet_copy_client.cc:523] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.189953 1065 tablet_bootstrap.cc:492] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:47.207307 1110 tablet_bootstrap.cc:492] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.207782 1110 tablet_bootstrap.cc:492] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap complete.
I20240501 20:10:47.208086 1110 ts_tablet_manager.cc:1367] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.055s user 0.044s sys 0.007s
I20240501 20:10:47.208904 1110 raft_consensus.cc:348] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:47.209139 1110 raft_consensus.cc:729] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:47.209424 1110 consensus_queue.cc:260] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:47.210880 1110 ts_tablet_manager.cc:1398] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:47.212078 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-10a4d24c5c844c8c9ccd191b1a2b0b00 received from {username='slave'} at 127.13.251.134:60949
I20240501 20:10:47.212213 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-10a4d24c5c844c8c9ccd191b1a2b0b00 on tablet 10a4d24c5c844c8c9ccd191b1a2b0b00 with peer c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:10:47.228971 1130 ts_tablet_manager.cc:902] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:47.243805 1065 tablet_bootstrap.cc:492] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.244246 1065 tablet_bootstrap.cc:492] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:10:47.244976 1065 ts_tablet_manager.cc:1367] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.016s
I20240501 20:10:47.246027 1065 raft_consensus.cc:348] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:47.246287 1065 raft_consensus.cc:729] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:47.246574 1065 consensus_queue.cc:260] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:47.247467 1065 ts_tablet_manager.cc:1398] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:47.249152 1130 tablet_copy_client.cc:308] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:10:47.249578 23734 tablet_copy_service.cc:342] P 51bf640e895b4e5eb91bdf7509ea7e46: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-46677b6d96bf45d98d061e572a85ebdc received from {username='slave'} at 127.13.251.131:50321
I20240501 20:10:47.249691 23734 tablet_copy_service.cc:433] P 51bf640e895b4e5eb91bdf7509ea7e46: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-46677b6d96bf45d98d061e572a85ebdc on tablet 46677b6d96bf45d98d061e572a85ebdc with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:10:47.250299 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 2ecad173c71744eca8ca537ad152f95f from peer 744c5a532e724391830c0785d32eb2fb ({username='slave'} at 127.13.251.135:46111)
I20240501 20:10:47.250430 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 2ecad173c71744eca8ca537ad152f95f from peer 744c5a532e724391830c0785d32eb2fb at {username='slave'} at 127.13.251.135:46111: session id = 744c5a532e724391830c0785d32eb2fb-2ecad173c71744eca8ca537ad152f95f
I20240501 20:10:47.253196 21378 tablet_copy_source_session.cc:205] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.258277 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecad173c71744eca8ca537ad152f95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.263355 1130 tablet_copy_client.cc:783] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.263736 1130 tablet_copy_client.cc:647] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.265323 1130 tablet_copy_client.cc:523] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.266783 21358 consensus_queue.cc:237] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:47.269181 1130 tablet_bootstrap.cc:492] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:47.269414 20962 raft_consensus.cc:1261] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.269992 21095 raft_consensus.cc:1261] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.270016 1022 consensus_queue.cc:1035] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:47.270586 21291 consensus_peers.cc:485] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): Couldn't send request to peer 744c5a532e724391830c0785d32eb2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 554b1d57dbb541508afaf8dbb77cbdad. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.271066 1022 consensus_queue.cc:1035] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.278705 21228 raft_consensus.cc:1203] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:47.289857 1022 raft_consensus.cc:2932] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:47.299427 21095 raft_consensus.cc:2932] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:47.299738 20962 raft_consensus.cc:2932] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:47.307298 20824 catalog_manager.cc:5484] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:47.309743 1053 ts_tablet_manager.cc:902] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:47.317230 1110 ts_tablet_manager.cc:902] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:10:47.319919 1130 tablet_bootstrap.cc:492] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.324534 1130 tablet_bootstrap.cc:492] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: Bootstrap complete.
I20240501 20:10:47.324864 1130 ts_tablet_manager.cc:1367] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.019s
I20240501 20:10:47.325732 1130 raft_consensus.cc:348] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:47.326035 1130 raft_consensus.cc:729] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:47.326300 1130 consensus_queue.cc:260] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:47.335829 1130 ts_tablet_manager.cc:1398] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:10:47.341612 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 744c5a532e724391830c0785d32eb2fb-2ecad173c71744eca8ca537ad152f95f received from {username='slave'} at 127.13.251.135:46111
I20240501 20:10:47.341825 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 744c5a532e724391830c0785d32eb2fb-2ecad173c71744eca8ca537ad152f95f on tablet 2ecad173c71744eca8ca537ad152f95f with peer 744c5a532e724391830c0785d32eb2fb
I20240501 20:10:47.347751 1053 tablet_copy_client.cc:308] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:10:47.348588 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet d75d2ef39a5347e2b6fbe3145c207280 from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:38225)
I20240501 20:10:47.348774 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet d75d2ef39a5347e2b6fbe3145c207280 from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:38225: session id = 51bf640e895b4e5eb91bdf7509ea7e46-d75d2ef39a5347e2b6fbe3145c207280
I20240501 20:10:47.352638 21248 tablet_copy_source_session.cc:205] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.354579 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75d2ef39a5347e2b6fbe3145c207280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.359915 1110 tablet_copy_client.cc:308] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:10:47.362399 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet 8d75b25758954877ab36e8029fb29463 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 ({username='slave'} at 127.13.251.134:60949)
I20240501 20:10:47.362552 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet 8d75b25758954877ab36e8029fb29463 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 at {username='slave'} at 127.13.251.134:60949: session id = c6dcaa8efff642dcb7c2c6e8be3a65f7-8d75b25758954877ab36e8029fb29463
I20240501 20:10:47.365190 21248 tablet_copy_source_session.cc:205] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.366452 1110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d75b25758954877ab36e8029fb29463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.369728 1053 tablet_copy_client.cc:783] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.370075 1053 tablet_copy_client.cc:647] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.373018 1053 tablet_copy_client.cc:523] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.373384 1110 tablet_copy_client.cc:783] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.373607 1110 tablet_copy_client.cc:647] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.375423 1110 tablet_copy_client.cc:523] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.377393 1053 tablet_bootstrap.cc:492] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:10:47.379444 1110 tablet_bootstrap.cc:492] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:47.398557 1053 tablet_bootstrap.cc:492] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.399134 1053 tablet_bootstrap.cc:492] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:10:47.399391 1053 ts_tablet_manager.cc:1367] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.006s
I20240501 20:10:47.400135 1053 raft_consensus.cc:348] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:47.400372 1053 raft_consensus.cc:729] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:10:47.400622 1053 consensus_queue.cc:260] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:47.401262 1053 ts_tablet_manager.cc:1398] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:47.402247 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-d75d2ef39a5347e2b6fbe3145c207280 received from {username='slave'} at 127.13.251.133:38225
I20240501 20:10:47.402349 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-d75d2ef39a5347e2b6fbe3145c207280 on tablet d75d2ef39a5347e2b6fbe3145c207280 with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:10:47.425901 21358 consensus_queue.cc:237] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:47.428792 21095 raft_consensus.cc:1261] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:47.429170 21294 consensus_peers.cc:485] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Couldn't send request to peer c6dcaa8efff642dcb7c2c6e8be3a65f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 998f71aeae984fd5b866cb866bd4e87a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.429432 1023 consensus_queue.cc:1035] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.430414 20962 raft_consensus.cc:1261] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.433086 1023 consensus_queue.cc:1035] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.435976 1110 tablet_bootstrap.cc:492] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.435688 1023 raft_consensus.cc:2932] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:47.436419 1110 tablet_bootstrap.cc:492] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap complete.
I20240501 20:10:47.436687 1110 ts_tablet_manager.cc:1367] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.058s user 0.049s sys 0.003s
I20240501 20:10:47.436420 20962 raft_consensus.cc:2932] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:47.437446 1110 raft_consensus.cc:348] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:47.437656 1110 raft_consensus.cc:729] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:47.437920 1110 consensus_queue.cc:260] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:47.438581 1110 ts_tablet_manager.cc:1398] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:10:47.439229 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-8d75b25758954877ab36e8029fb29463 received from {username='slave'} at 127.13.251.134:60949
I20240501 20:10:47.439330 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-8d75b25758954877ab36e8029fb29463 on tablet 8d75b25758954877ab36e8029fb29463 with peer c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:10:47.441128 21095 raft_consensus.cc:2932] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:47.444039 20824 catalog_manager.cc:5484] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:47.450685 1065 ts_tablet_manager.cc:902] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:47.464454 1065 tablet_copy_client.cc:308] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:41259
I20240501 20:10:47.465112 20987 tablet_copy_service.cc:140] P cda19d6a81c0473db302535ce2ada7f0: Received BeginTabletCopySession request for tablet c018d1b99ad749e59a2cf338001d38e4 from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:41291)
I20240501 20:10:47.465265 20987 tablet_copy_service.cc:161] P cda19d6a81c0473db302535ce2ada7f0: Beginning new tablet copy session on tablet c018d1b99ad749e59a2cf338001d38e4 from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:41291: session id = 4af096f5a6a9440a94f6682b399efe20-c018d1b99ad749e59a2cf338001d38e4
I20240501 20:10:47.467931 20987 tablet_copy_source_session.cc:205] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.469122 1065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c018d1b99ad749e59a2cf338001d38e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.472975 23709 raft_consensus.cc:1203] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:47.479321 1065 tablet_copy_client.cc:783] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.479635 1065 tablet_copy_client.cc:647] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.481206 1065 tablet_copy_client.cc:523] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.502564 1065 tablet_bootstrap.cc:492] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:47.513671 1192 ts_tablet_manager.cc:902] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:47.518252 1065 tablet_bootstrap.cc:492] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.518633 1065 tablet_bootstrap.cc:492] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:10:47.518883 1065 ts_tablet_manager.cc:1367] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.001s
I20240501 20:10:47.520269 1065 raft_consensus.cc:348] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:47.520507 1065 raft_consensus.cc:729] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:47.520992 1065 consensus_queue.cc:260] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:47.524076 1065 ts_tablet_manager.cc:1398] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:10:47.524765 20987 tablet_copy_service.cc:342] P cda19d6a81c0473db302535ce2ada7f0: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-c018d1b99ad749e59a2cf338001d38e4 received from {username='slave'} at 127.13.251.131:41291
I20240501 20:10:47.524869 20987 tablet_copy_service.cc:433] P cda19d6a81c0473db302535ce2ada7f0: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-c018d1b99ad749e59a2cf338001d38e4 on tablet c018d1b99ad749e59a2cf338001d38e4 with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:10:47.544049 1192 tablet_copy_client.cc:308] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:10:47.561771 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 748b4fd500f84a359bd8ee05b861300d from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:58619)
I20240501 20:10:47.561950 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 748b4fd500f84a359bd8ee05b861300d from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:58619: session id = 8559229e044b4e1ea661b881b3f9af50-748b4fd500f84a359bd8ee05b861300d
I20240501 20:10:47.564857 21378 tablet_copy_source_session.cc:205] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.573518 1192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748b4fd500f84a359bd8ee05b861300d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.584425 23975 raft_consensus.cc:1203] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:47.590981 1192 tablet_copy_client.cc:783] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.592638 1192 tablet_copy_client.cc:647] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.594295 1192 tablet_copy_client.cc:523] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.599264 1192 tablet_bootstrap.cc:492] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:47.646236 21225 raft_consensus.cc:1203] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:47.659122 23846 raft_consensus.cc:1203] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:47.664053 1192 tablet_bootstrap.cc:492] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.664520 1192 tablet_bootstrap.cc:492] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:10:47.665102 1192 ts_tablet_manager.cc:1367] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.066s user 0.046s sys 0.009s
I20240501 20:10:47.666224 1192 raft_consensus.cc:348] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:47.666469 1192 raft_consensus.cc:729] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:47.666747 1192 consensus_queue.cc:260] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:47.668387 1192 ts_tablet_manager.cc:1398] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:47.669629 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-748b4fd500f84a359bd8ee05b861300d received from {username='slave'} at 127.13.251.136:58619
I20240501 20:10:47.669778 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-748b4fd500f84a359bd8ee05b861300d on tablet 748b4fd500f84a359bd8ee05b861300d with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:10:47.690459 1065 ts_tablet_manager.cc:902] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:47.712796 1065 tablet_copy_client.cc:308] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:10:47.715678 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet 2b57a80759944301a033140e482aec36 from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:54857)
I20240501 20:10:47.715867 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet 2b57a80759944301a033140e482aec36 from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:54857: session id = 4af096f5a6a9440a94f6682b399efe20-2b57a80759944301a033140e482aec36
I20240501 20:10:47.718896 23866 tablet_copy_source_session.cc:205] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.720273 1065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b57a80759944301a033140e482aec36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.726734 1065 tablet_copy_client.cc:783] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.727025 1065 tablet_copy_client.cc:647] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.742807 1065 tablet_copy_client.cc:523] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.747833 1065 tablet_bootstrap.cc:492] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:10:47.759011 23846 raft_consensus.cc:1203] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:47.766371 1130 ts_tablet_manager.cc:902] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:47.770150 23710 raft_consensus.cc:1203] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:47.775899 1065 tablet_bootstrap.cc:492] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.776340 1065 tablet_bootstrap.cc:492] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:10:47.776715 1065 ts_tablet_manager.cc:1367] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.002s
I20240501 20:10:47.777588 1065 raft_consensus.cc:348] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:47.777830 1065 raft_consensus.cc:729] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:10:47.778079 1065 consensus_queue.cc:260] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:47.778795 1065 ts_tablet_manager.cc:1398] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:47.781828 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-2b57a80759944301a033140e482aec36 received from {username='slave'} at 127.13.251.131:54857
I20240501 20:10:47.781965 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-2b57a80759944301a033140e482aec36 on tablet 2b57a80759944301a033140e482aec36 with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:10:47.803071 1130 tablet_copy_client.cc:308] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:10:47.803706 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 554b1d57dbb541508afaf8dbb77cbdad from peer 744c5a532e724391830c0785d32eb2fb ({username='slave'} at 127.13.251.135:46111)
I20240501 20:10:47.803885 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 554b1d57dbb541508afaf8dbb77cbdad from peer 744c5a532e724391830c0785d32eb2fb at {username='slave'} at 127.13.251.135:46111: session id = 744c5a532e724391830c0785d32eb2fb-554b1d57dbb541508afaf8dbb77cbdad
I20240501 20:10:47.806674 21378 tablet_copy_source_session.cc:205] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.810751 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554b1d57dbb541508afaf8dbb77cbdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.836512 1130 tablet_copy_client.cc:783] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.836760 1130 tablet_copy_client.cc:647] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.842983 1130 tablet_copy_client.cc:523] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.851706 1130 tablet_bootstrap.cc:492] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: Bootstrap starting.
I20240501 20:10:47.858139 1110 ts_tablet_manager.cc:902] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:10:47.941293 24109 raft_consensus.cc:1203] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:47.957890 1110 tablet_copy_client.cc:308] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:10:47.958516 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 998f71aeae984fd5b866cb866bd4e87a from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 ({username='slave'} at 127.13.251.134:57127)
I20240501 20:10:47.958688 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 998f71aeae984fd5b866cb866bd4e87a from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 at {username='slave'} at 127.13.251.134:57127: session id = c6dcaa8efff642dcb7c2c6e8be3a65f7-998f71aeae984fd5b866cb866bd4e87a
I20240501 20:10:47.961511 21378 tablet_copy_source_session.cc:205] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.964711 1110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998f71aeae984fd5b866cb866bd4e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.966621 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:47.972766 1110 tablet_copy_client.cc:783] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.973007 1110 tablet_copy_client.cc:647] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.985112 1110 tablet_copy_client.cc:523] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.991468 1110 tablet_bootstrap.cc:492] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:10:47.997411 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:47.997609 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:48.027050 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:48.053998 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:48.059693 21227 raft_consensus.cc:1203] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:48.146621 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:48.171823 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:48.211875 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:48.244450 1110 tablet_bootstrap.cc:492] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:48.245013 1110 tablet_bootstrap.cc:492] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap complete.
I20240501 20:10:48.245288 1110 ts_tablet_manager.cc:1367] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.254s user 0.066s sys 0.008s
I20240501 20:10:48.246728 1110 raft_consensus.cc:348] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:48.246944 1110 raft_consensus.cc:729] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:10:48.247200 1110 consensus_queue.cc:260] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:48.249445 1110 ts_tablet_manager.cc:1398] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:10:48.256317 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-998f71aeae984fd5b866cb866bd4e87a received from {username='slave'} at 127.13.251.134:57127
I20240501 20:10:48.256470 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-998f71aeae984fd5b866cb866bd4e87a on tablet 998f71aeae984fd5b866cb866bd4e87a with peer c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:10:48.265547 1130 tablet_bootstrap.cc:492] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:48.265952 1130 tablet_bootstrap.cc:492] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: Bootstrap complete.
I20240501 20:10:48.266222 1130 ts_tablet_manager.cc:1367] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: Time spent bootstrapping tablet: real 0.415s user 0.056s sys 0.000s
I20240501 20:10:48.266940 1130 raft_consensus.cc:348] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:48.267159 1130 raft_consensus.cc:729] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744c5a532e724391830c0785d32eb2fb, State: Initialized, Role: FOLLOWER
I20240501 20:10:48.267369 1130 consensus_queue.cc:260] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:48.278864 1130 ts_tablet_manager.cc:1398] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:10:48.281582 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 744c5a532e724391830c0785d32eb2fb-554b1d57dbb541508afaf8dbb77cbdad received from {username='slave'} at 127.13.251.135:46111
I20240501 20:10:48.281739 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 744c5a532e724391830c0785d32eb2fb-554b1d57dbb541508afaf8dbb77cbdad on tablet 554b1d57dbb541508afaf8dbb77cbdad with peer 744c5a532e724391830c0785d32eb2fb
I20240501 20:10:48.281992 23974 raft_consensus.cc:1203] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:48.283227 21227 raft_consensus.cc:1203] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:48.283330 23843 raft_consensus.cc:1203] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:48.759469 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:48.759652 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:48.794466 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:48.812083 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:10:48.844310 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 44291.1us
I20240501 20:10:48.844487 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:10:48.872817 20901 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 98566us
I20240501 20:10:48.884693 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:48.885246 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:48.925750 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:49.284907 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:49.301296 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:49.313928 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:49.342500 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:49.412930 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:49.459977 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:49.462831 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240501 20:10:49.558418 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 59919.5us
I20240501 20:10:49.586772 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:49.715677 21357 consensus_queue.cc:237] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:49.717687 24109 raft_consensus.cc:1261] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:49.717664 21227 raft_consensus.cc:1261] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:49.718304 1404 consensus_queue.cc:1035] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:49.718591 1405 consensus_queue.cc:1035] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:49.721511 1404 raft_consensus.cc:2932] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:49.722764 24109 raft_consensus.cc:2932] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:49.725975 21227 raft_consensus.cc:2932] T 748b4fd500f84a359bd8ee05b861300d P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:49.730801 20825 catalog_manager.cc:5484] T 748b4fd500f84a359bd8ee05b861300d P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:49.736138 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet 748b4fd500f84a359bd8ee05b861300d with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51330
I20240501 20:10:49.737630 1433 tablet_replica.cc:330] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:10:49.737846 1433 raft_consensus.cc:2229] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:49.738133 1433 raft_consensus.cc:2258] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:49.738835 1433 ts_tablet_manager.cc:1875] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:49.745343 1433 ts_tablet_manager.cc:1888] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:49.745572 1433 log.cc:1196] T 748b4fd500f84a359bd8ee05b861300d P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/748b4fd500f84a359bd8ee05b861300d
I20240501 20:10:49.746403 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 748b4fd500f84a359bd8ee05b861300d (table rebalance_test_table_15 [id=4e06f1256f6f44669a58b9283a607e57]) successfully deleted
I20240501 20:10:49.971239 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:49.989984 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:49.998292 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:50.057667 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:50.093606 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:50.101855 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:50.116283 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:50.201787 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:50.344435 21227 consensus_queue.cc:237] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:50.346251 21098 raft_consensus.cc:1261] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:50.346318 23843 raft_consensus.cc:1261] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:50.346776 685 consensus_queue.cc:1035] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:50.347021 1296 consensus_queue.cc:1035] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:50.349357 685 raft_consensus.cc:2932] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:50.350132 21098 raft_consensus.cc:2932] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:50.359359 20824 catalog_manager.cc:5484] T 8d75b25758954877ab36e8029fb29463 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:50.366907 23843 raft_consensus.cc:2932] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:50.374915 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d75b25758954877ab36e8029fb29463 with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42940
I20240501 20:10:50.382046 1518 tablet_replica.cc:330] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:10:50.382334 1518 raft_consensus.cc:2229] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:50.382649 1518 raft_consensus.cc:2258] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:50.383673 1518 ts_tablet_manager.cc:1875] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:50.391654 1518 ts_tablet_manager.cc:1888] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:50.391777 1518 log.cc:1196] T 8d75b25758954877ab36e8029fb29463 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/8d75b25758954877ab36e8029fb29463
I20240501 20:10:50.392644 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 8d75b25758954877ab36e8029fb29463 (table rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]) successfully deleted
I20240501 20:10:50.740712 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:50.760362 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:50.766028 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:50.769485 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:50.797417 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:50.844923 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:50.849967 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:50.891515 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:51.058051 20962 consensus_queue.cc:237] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:51.059868 21357 raft_consensus.cc:1261] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:51.060369 21227 raft_consensus.cc:1261] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:51.060348 1401 consensus_queue.cc:1035] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:51.060876 1528 consensus_queue.cc:1035] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:51.073239 1614 raft_consensus.cc:2932] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:51.075829 21227 raft_consensus.cc:2932] T c018d1b99ad749e59a2cf338001d38e4 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:51.076009 21357 raft_consensus.cc:2932] T c018d1b99ad749e59a2cf338001d38e4 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:51.080001 20820 catalog_manager.cc:5484] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index 2 to 3, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:51.116844 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet c018d1b99ad749e59a2cf338001d38e4 with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51330
I20240501 20:10:51.126886 1623 tablet_replica.cc:330] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:10:51.127094 1623 raft_consensus.cc:2229] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:51.127250 1623 raft_consensus.cc:2258] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:51.127990 1623 ts_tablet_manager.cc:1875] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:51.137599 1623 ts_tablet_manager.cc:1888] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:51.137840 1623 log.cc:1196] T c018d1b99ad749e59a2cf338001d38e4 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/c018d1b99ad749e59a2cf338001d38e4
I20240501 20:10:51.138969 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet c018d1b99ad749e59a2cf338001d38e4 (table rebalancer_extra_table_42 [id=f29d050ce033498db24871da9ff2efa8]) successfully deleted
I20240501 20:10:51.428597 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:51.446699 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:51.458740 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:51.465585 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:51.525756 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:51.537680 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:10:51.558738 21161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 47250us
I20240501 20:10:51.593917 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:51.653741 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:51.795867 21357 consensus_queue.cc:237] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:51.798347 24109 raft_consensus.cc:1261] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:51.798395 23975 raft_consensus.cc:1261] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:51.798938 1404 consensus_queue.cc:1035] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:51.799237 1682 consensus_queue.cc:1035] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:51.802783 1404 raft_consensus.cc:2932] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:51.803766 23975 raft_consensus.cc:2932] T 2ecad173c71744eca8ca537ad152f95f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:51.805832 24109 raft_consensus.cc:2932] T 2ecad173c71744eca8ca537ad152f95f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:51.810459 20819 catalog_manager.cc:5484] T 2ecad173c71744eca8ca537ad152f95f P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:51.828029 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecad173c71744eca8ca537ad152f95f with delete_type TABLET_DATA_TOMBSTONED (TS cda19d6a81c0473db302535ce2ada7f0 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54238
I20240501 20:10:51.832056 1711 tablet_replica.cc:330] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:10:51.832605 1711 raft_consensus.cc:2229] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:51.832902 1711 raft_consensus.cc:2258] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:51.833513 1711 ts_tablet_manager.cc:1875] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:51.839929 1711 ts_tablet_manager.cc:1888] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:51.840196 1711 log.cc:1196] T 2ecad173c71744eca8ca537ad152f95f P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ecad173c71744eca8ca537ad152f95f
I20240501 20:10:51.841434 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 2ecad173c71744eca8ca537ad152f95f (table rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]) successfully deleted
I20240501 20:10:52.248230 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:52.255925 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:52.284873 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:52.285811 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:52.354964 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:52.355437 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:52.380323 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:52.433074 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:52.617754 21225 consensus_queue.cc:237] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:52.620064 23710 raft_consensus.cc:1261] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:52.620412 21096 raft_consensus.cc:1261] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:52.620680 1772 consensus_queue.cc:1035] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:52.620921 1512 consensus_queue.cc:1035] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:52.623510 1512 raft_consensus.cc:2932] T d75d2ef39a5347e2b6fbe3145c207280 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:52.624927 23710 raft_consensus.cc:2932] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:52.625087 21096 raft_consensus.cc:2932] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:52.631002 20825 catalog_manager.cc:5484] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:52.645164 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet d75d2ef39a5347e2b6fbe3145c207280 with delete_type TABLET_DATA_TOMBSTONED (TS 8559229e044b4e1ea661b881b3f9af50 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:52.661310 1795 tablet_replica.cc:330] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:52.661759 1795 raft_consensus.cc:2229] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:52.662091 1795 raft_consensus.cc:2258] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:52.663013 1795 ts_tablet_manager.cc:1875] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:52.669598 1795 ts_tablet_manager.cc:1888] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:52.669732 1795 log.cc:1196] T d75d2ef39a5347e2b6fbe3145c207280 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/d75d2ef39a5347e2b6fbe3145c207280
I20240501 20:10:52.670468 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet d75d2ef39a5347e2b6fbe3145c207280 (table rebalancer_extra_table_40 [id=fa5b0afd37a145e9b7cf26f65340816f]) successfully deleted
I20240501 20:10:52.891690 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:52.923725 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:52.934593 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:52.943785 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:52.980994 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:53.028677 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:53.043789 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:53.064816 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:53.226377 21225 consensus_queue.cc:237] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:10:53.228407 20962 raft_consensus.cc:1261] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:53.228925 1772 consensus_queue.cc:1035] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:53.230078 23846 raft_consensus.cc:1261] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:53.230660 1772 consensus_queue.cc:1035] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:53.232231 1512 raft_consensus.cc:2932] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:53.232825 20962 raft_consensus.cc:2932] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:53.234172 23846 raft_consensus.cc:2932] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:53.238530 20817 catalog_manager.cc:5484] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:10:53.246130 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet 10a4d24c5c844c8c9ccd191b1a2b0b00 with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42940
I20240501 20:10:53.247689 1878 tablet_replica.cc:330] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:10:53.247977 1878 raft_consensus.cc:2229] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:53.248268 1878 raft_consensus.cc:2258] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:53.249234 1878 ts_tablet_manager.cc:1875] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:53.256657 1878 ts_tablet_manager.cc:1888] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:53.256834 1878 log.cc:1196] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/10a4d24c5c844c8c9ccd191b1a2b0b00
I20240501 20:10:53.257669 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 10a4d24c5c844c8c9ccd191b1a2b0b00 (table rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]) successfully deleted
I20240501 20:10:53.589726 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:53.636828 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:53.648861 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:53.692780 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:53.784901 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:53.810084 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:53.810788 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:53.829128 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:53.950249 21358 consensus_queue.cc:237] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:10:53.952927 23974 raft_consensus.cc:1261] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:53.952945 21096 raft_consensus.cc:1261] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:53.953465 1404 consensus_queue.cc:1035] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:53.953768 1682 consensus_queue.cc:1035] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:53.956578 1682 raft_consensus.cc:2932] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:53.959354 21096 raft_consensus.cc:2932] T 554b1d57dbb541508afaf8dbb77cbdad P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:53.964555 20825 catalog_manager.cc:5484] T 554b1d57dbb541508afaf8dbb77cbdad P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:53.970142 23974 raft_consensus.cc:2932] T 554b1d57dbb541508afaf8dbb77cbdad P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:10:53.986356 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 554b1d57dbb541508afaf8dbb77cbdad with delete_type TABLET_DATA_TOMBSTONED (TS cda19d6a81c0473db302535ce2ada7f0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54238
I20240501 20:10:53.991428 1967 tablet_replica.cc:330] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:10:53.991693 1967 raft_consensus.cc:2229] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:53.991945 1967 raft_consensus.cc:2258] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:53.992687 1967 ts_tablet_manager.cc:1875] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:53.998461 1967 ts_tablet_manager.cc:1888] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:53.998642 1967 log.cc:1196] T 554b1d57dbb541508afaf8dbb77cbdad P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/554b1d57dbb541508afaf8dbb77cbdad
I20240501 20:10:54.004839 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 554b1d57dbb541508afaf8dbb77cbdad (table rebalance_test_table_7 [id=7ac9725542f64c3ea5f268eab72de052]) successfully deleted
I20240501 20:10:54.314379 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:54.339036 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:54.355885 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:54.359488 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:54.369379 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:54.370563 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:54.403815 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:54.460529 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:54.630278 23846 consensus_queue.cc:237] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:54.645529 21227 raft_consensus.cc:1261] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:54.645932 21356 raft_consensus.cc:1261] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:54.646095 2023 consensus_queue.cc:1035] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:54.646446 1591 consensus_queue.cc:1035] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:54.649395 1591 raft_consensus.cc:2932] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:54.650303 21356 raft_consensus.cc:2932] T a7498fc5cf5144d58712eb2c8f188c5e P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:54.653461 21227 raft_consensus.cc:2932] T a7498fc5cf5144d58712eb2c8f188c5e P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:54.659030 20820 catalog_manager.cc:5484] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 2 to 3, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:54.666471 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet a7498fc5cf5144d58712eb2c8f188c5e with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51330
I20240501 20:10:54.667083 2046 tablet_replica.cc:330] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:10:54.667397 2046 raft_consensus.cc:2229] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:54.668165 2046 raft_consensus.cc:2258] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:54.669642 2046 ts_tablet_manager.cc:1875] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:54.674607 2046 ts_tablet_manager.cc:1888] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:54.674717 2046 log.cc:1196] T a7498fc5cf5144d58712eb2c8f188c5e P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/a7498fc5cf5144d58712eb2c8f188c5e
I20240501 20:10:54.675652 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet a7498fc5cf5144d58712eb2c8f188c5e (table rebalancer_extra_table_48 [id=9d8ae33f4ae246d999678c652a49b2cc]) successfully deleted
I20240501 20:10:54.917346 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:54.947567 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:54.986542 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:54.995069 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:55.032291 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:55.033241 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:55.082319 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:55.091248 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:55.265065 20963 consensus_queue.cc:237] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:10:55.267329 23710 raft_consensus.cc:1261] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:55.267396 21227 raft_consensus.cc:1261] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:55.267918 1401 consensus_queue.cc:1035] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:55.268222 1614 consensus_queue.cc:1035] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:55.270762 1401 raft_consensus.cc:2932] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:55.271464 21227 raft_consensus.cc:2932] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:55.276374 23710 raft_consensus.cc:2932] T 73fcc9ac412643088c996b76fbb1e680 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:55.278774 20824 catalog_manager.cc:5484] T 73fcc9ac412643088c996b76fbb1e680 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 2 to 3, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:10:55.322193 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 73fcc9ac412643088c996b76fbb1e680 with delete_type TABLET_DATA_TOMBSTONED (TS 8559229e044b4e1ea661b881b3f9af50 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:55.326252 2130 tablet_replica.cc:330] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:55.326583 2130 raft_consensus.cc:2229] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:55.326836 2130 raft_consensus.cc:2258] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:55.327461 2130 ts_tablet_manager.cc:1875] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:55.336551 2130 ts_tablet_manager.cc:1888] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:55.336736 2130 log.cc:1196] T 73fcc9ac412643088c996b76fbb1e680 P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/73fcc9ac412643088c996b76fbb1e680
I20240501 20:10:55.337877 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 73fcc9ac412643088c996b76fbb1e680 (table rebalancer_extra_table_44 [id=ada4caf0639449479d1aa6194393f5b8]) successfully deleted
I20240501 20:10:55.607300 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:55.617699 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:55.644295 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:55.705562 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:55.733294 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:55.750113 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:55.764554 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:55.804127 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:55.959493 23842 consensus_queue.cc:237] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:55.961723 21227 raft_consensus.cc:1261] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:55.962270 24108 raft_consensus.cc:1261] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:55.962332 2023 consensus_queue.cc:1035] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:55.962893 1591 consensus_queue.cc:1035] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:55.965973 1591 raft_consensus.cc:2932] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:55.966814 24108 raft_consensus.cc:2932] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:55.969543 21227 raft_consensus.cc:2932] T 2b57a80759944301a033140e482aec36 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:55.973342 20818 catalog_manager.cc:5484] T 2b57a80759944301a033140e482aec36 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:55.987396 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b57a80759944301a033140e482aec36 with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:55.994074 2208 tablet_replica.cc:330] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:55.996969 2208 raft_consensus.cc:2229] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:55.997339 2208 raft_consensus.cc:2258] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:55.998013 2208 ts_tablet_manager.cc:1875] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:56.004309 2208 ts_tablet_manager.cc:1888] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:56.004539 2208 log.cc:1196] T 2b57a80759944301a033140e482aec36 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/2b57a80759944301a033140e482aec36
I20240501 20:10:56.005683 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 2b57a80759944301a033140e482aec36 (table rebalancer_extra_table_18 [id=2ff44c377d82487fa882622ec97d33eb]) successfully deleted
I20240501 20:10:56.302006 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:56.317543 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:56.320875 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:56.351588 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:56.360564 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:56.399791 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:56.422032 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:56.452889 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:56.644649 23710 consensus_queue.cc:237] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:56.647264 23975 raft_consensus.cc:1261] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:56.647434 21227 raft_consensus.cc:1261] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:56.647817 1132 consensus_queue.cc:1035] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:56.648064 1975 consensus_queue.cc:1035] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:56.651089 1975 raft_consensus.cc:2932] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:56.651790 23975 raft_consensus.cc:2932] T 46677b6d96bf45d98d061e572a85ebdc P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:56.652148 21227 raft_consensus.cc:2932] T 46677b6d96bf45d98d061e572a85ebdc P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:56.658365 20822 catalog_manager.cc:5484] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 3, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) evicted. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:56.666378 20945 tablet_service.cc:1514] Processing DeleteTablet for tablet 46677b6d96bf45d98d061e572a85ebdc with delete_type TABLET_DATA_TOMBSTONED (TS cda19d6a81c0473db302535ce2ada7f0 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54238
I20240501 20:10:56.667066 2301 tablet_replica.cc:330] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0: stopping tablet replica
I20240501 20:10:56.667276 2301 raft_consensus.cc:2229] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:56.667565 2301 raft_consensus.cc:2258] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:56.668239 2301 ts_tablet_manager.cc:1875] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:56.675295 2301 ts_tablet_manager.cc:1888] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:56.675411 2301 log.cc:1196] T 46677b6d96bf45d98d061e572a85ebdc P cda19d6a81c0473db302535ce2ada7f0: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-0/wal/wals/46677b6d96bf45d98d061e572a85ebdc
I20240501 20:10:56.676278 20810 catalog_manager.cc:4830] TS cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): tablet 46677b6d96bf45d98d061e572a85ebdc (table rebalancer_extra_table_46 [id=8176cb275a1b486f8ff0517afba67297]) successfully deleted
I20240501 20:10:56.931108 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:56.935916 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:56.953997 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:56.979337 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:56.988435 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:57.002985 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:57.058943 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:57.075640 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:57.146634 20815 maintenance_manager.cc:419] P f878915f05c04c2c8c7cd9cf39cd6cf8: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033386
I20240501 20:10:57.254482 20814 maintenance_manager.cc:643] P f878915f05c04c2c8c7cd9cf39cd6cf8: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.107s user 0.073s sys 0.004s Metrics: {"bytes_written":93993,"cfile_init":1,"dirs.queue_time_us":370,"dirs.run_cpu_time_us":287,"dirs.run_wall_time_us":1113,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1631,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":257,"thread_start_us":243,"threads_started":1}
I20240501 20:10:57.256386 20815 maintenance_manager.cc:419] P f878915f05c04c2c8c7cd9cf39cd6cf8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 30268 bytes on disk
I20240501 20:10:57.257301 20814 maintenance_manager.cc:643] P f878915f05c04c2c8c7cd9cf39cd6cf8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 20:10:57.273582 24107 consensus_queue.cc:237] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:10:57.275715 21227 raft_consensus.cc:1261] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:57.276494 1491 consensus_queue.cc:1035] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:57.276587 20963 raft_consensus.cc:1261] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:57.277354 1491 consensus_queue.cc:1035] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:57.281538 1855 raft_consensus.cc:2932] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:57.282243 21227 raft_consensus.cc:2932] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:57.288201 20963 raft_consensus.cc:2932] T 3f3951bb18524e41aa6836122e97e7f5 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:57.288987 20823 catalog_manager.cc:5484] T 3f3951bb18524e41aa6836122e97e7f5 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:10:57.316931 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3951bb18524e41aa6836122e97e7f5 with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51874
I20240501 20:10:57.330453 2386 tablet_replica.cc:330] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:10:57.330807 2386 raft_consensus.cc:2229] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:57.331615 2386 raft_consensus.cc:2258] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:57.332449 2386 ts_tablet_manager.cc:1875] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:57.339975 2386 ts_tablet_manager.cc:1888] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:10:57.340189 2386 log.cc:1196] T 3f3951bb18524e41aa6836122e97e7f5 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/3f3951bb18524e41aa6836122e97e7f5
I20240501 20:10:57.340924 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 3f3951bb18524e41aa6836122e97e7f5 (table rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]) successfully deleted
I20240501 20:10:57.604190 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:57.676172 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:10:57.693959 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 62757.4us
I20240501 20:10:57.701124 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:57.707835 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:57.708257 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:57.719257 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:57.729363 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:57.769626 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:57.914754 23975 consensus_queue.cc:237] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:10:57.916769 21357 raft_consensus.cc:1261] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:57.916831 23846 raft_consensus.cc:1261] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:57.917383 1766 consensus_queue.cc:1035] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:57.917706 2446 consensus_queue.cc:1035] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:57.920094 1766 raft_consensus.cc:2932] T 30ea740ef5534b089c77063a8d8b9aaa P 744c5a532e724391830c0785d32eb2fb [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:57.922328 21357 raft_consensus.cc:2932] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:57.929230 20823 catalog_manager.cc:5484] T 30ea740ef5534b089c77063a8d8b9aaa P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 2 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New cstate: current_term: 2 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:57.930728 23846 raft_consensus.cc:2932] T 30ea740ef5534b089c77063a8d8b9aaa P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:10:57.942629 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ea740ef5534b089c77063a8d8b9aaa with delete_type TABLET_DATA_TOMBSTONED (TS 8559229e044b4e1ea661b881b3f9af50 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49700
I20240501 20:10:57.953982 2468 tablet_replica.cc:330] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: stopping tablet replica
I20240501 20:10:57.954342 2468 raft_consensus.cc:2229] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:57.955861 2468 raft_consensus.cc:2258] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:57.956821 2468 ts_tablet_manager.cc:1875] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:57.976275 2468 ts_tablet_manager.cc:1888] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 20:10:57.987679 2468 log.cc:1196] T 30ea740ef5534b089c77063a8d8b9aaa P 8559229e044b4e1ea661b881b3f9af50: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-7/wal/wals/30ea740ef5534b089c77063a8d8b9aaa
I20240501 20:10:57.988667 20812 catalog_manager.cc:4830] TS 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): tablet 30ea740ef5534b089c77063a8d8b9aaa (table rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]) successfully deleted
I20240501 20:10:58.131100 23846 consensus_queue.cc:237] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:58.133554 23709 raft_consensus.cc:1261] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:58.133800 20963 raft_consensus.cc:1261] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:58.134052 2444 consensus_queue.cc:1035] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:58.134378 2023 consensus_queue.cc:1035] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:58.134683 23779 consensus_peers.cc:485] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74f23f1dafeb4447b33cb7f5bdbf03c4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:58.137614 2444 raft_consensus.cc:2932] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:58.138243 20963 raft_consensus.cc:2932] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:58.142623 23709 raft_consensus.cc:2932] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:58.145769 20826 catalog_manager.cc:5484] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:58.193665 23975 consensus_queue.cc:237] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:58.196763 21098 raft_consensus.cc:1261] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:58.196942 23846 raft_consensus.cc:1261] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:58.197458 1766 consensus_queue.cc:1035] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:58.197876 2446 consensus_queue.cc:1035] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:58.198487 23909 consensus_peers.cc:485] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23bf7844614647fa86b23155bf69839d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:58.201570 2359 raft_consensus.cc:2932] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:58.202312 21098 raft_consensus.cc:2932] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:58.206576 23846 raft_consensus.cc:2932] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:58.213786 20823 catalog_manager.cc:5484] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:58.262425 20963 consensus_queue.cc:237] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:58.264824 21098 raft_consensus.cc:1261] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:58.265357 2549 consensus_queue.cc:1035] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:58.265799 21357 raft_consensus.cc:1261] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:58.266291 2549 consensus_queue.cc:1035] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:58.266570 20902 consensus_peers.cc:485] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75a1b78bb9574b588e6bf69ae7e68bc3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:58.269579 2342 raft_consensus.cc:2932] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:58.270143 21096 raft_consensus.cc:2932] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:58.277633 20824 catalog_manager.cc:5484] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:58.279374 21357 raft_consensus.cc:2932] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:10:58.332188 23975 consensus_queue.cc:237] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:58.334923 23709 raft_consensus.cc:1261] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:58.335285 21357 raft_consensus.cc:1261] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:58.335345 23909 consensus_peers.cc:485] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb -> Peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): Couldn't send request to peer feac73db0a3b4aafa374ca5dfd322b08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 654895cdb221479da79da05bc1938876. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:58.336520 2532 consensus_queue.cc:1035] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:58.337584 1766 consensus_queue.cc:1035] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:58.340667 2532 raft_consensus.cc:2932] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:58.341250 21357 raft_consensus.cc:2932] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:58.344880 23709 raft_consensus.cc:2932] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:58.347998 20826 catalog_manager.cc:5484] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index -1 to 2, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) added. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:58.391841 24108 consensus_queue.cc:237] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:58.394512 23709 raft_consensus.cc:1261] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:58.394722 23842 raft_consensus.cc:1261] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:58.395059 24043 consensus_peers.cc:485] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d2e66020ffb438fad69c2ea73762aa7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:58.395098 2586 consensus_queue.cc:1035] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:58.395408 1855 consensus_queue.cc:1035] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:58.397946 2586 raft_consensus.cc:2932] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:58.400921 23709 raft_consensus.cc:2932] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:58.401810 23842 raft_consensus.cc:2932] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:58.408370 20826 catalog_manager.cc:5484] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:10:58.619993 2623 ts_tablet_manager.cc:902] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:10:58.691694 2623 tablet_copy_client.cc:308] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:10:58.692641 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet 74f23f1dafeb4447b33cb7f5bdbf03c4 from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:58461)
I20240501 20:10:58.692814 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet 74f23f1dafeb4447b33cb7f5bdbf03c4 from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:58461: session id = feac73db0a3b4aafa374ca5dfd322b08-74f23f1dafeb4447b33cb7f5bdbf03c4
I20240501 20:10:58.696959 23866 tablet_copy_source_session.cc:205] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:58.704399 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f23f1dafeb4447b33cb7f5bdbf03c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.719893 2623 tablet_copy_client.cc:783] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:58.720165 2623 tablet_copy_client.cc:647] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:58.730687 2640 ts_tablet_manager.cc:902] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:10:58.737121 2623 tablet_copy_client.cc:523] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:58.741585 2623 tablet_bootstrap.cc:492] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:58.764317 2623 tablet_bootstrap.cc:492] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:58.767860 2623 tablet_bootstrap.cc:492] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:10:58.770176 2623 ts_tablet_manager.cc:1367] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.005s
I20240501 20:10:58.774139 2623 raft_consensus.cc:348] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:58.774818 2623 raft_consensus.cc:729] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.775321 2623 consensus_queue.cc:260] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:58.776597 2623 ts_tablet_manager.cc:1398] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:58.777787 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-74f23f1dafeb4447b33cb7f5bdbf03c4 received from {username='slave'} at 127.13.251.130:58461
I20240501 20:10:58.777992 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-74f23f1dafeb4447b33cb7f5bdbf03c4 on tablet 74f23f1dafeb4447b33cb7f5bdbf03c4 with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:10:58.806553 2642 ts_tablet_manager.cc:902] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:58.815428 2647 ts_tablet_manager.cc:902] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:10:58.837136 2640 tablet_copy_client.cc:308] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:41259
I20240501 20:10:58.837860 20987 tablet_copy_service.cc:140] P cda19d6a81c0473db302535ce2ada7f0: Received BeginTabletCopySession request for tablet 75a1b78bb9574b588e6bf69ae7e68bc3 from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:35157)
I20240501 20:10:58.838104 20987 tablet_copy_service.cc:161] P cda19d6a81c0473db302535ce2ada7f0: Beginning new tablet copy session on tablet 75a1b78bb9574b588e6bf69ae7e68bc3 from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:35157: session id = 8559229e044b4e1ea661b881b3f9af50-75a1b78bb9574b588e6bf69ae7e68bc3
I20240501 20:10:58.846691 20987 tablet_copy_source_session.cc:205] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:58.848429 2640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a1b78bb9574b588e6bf69ae7e68bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.853754 2640 tablet_copy_client.cc:783] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:58.853958 2640 tablet_copy_client.cc:647] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:58.856478 2642 tablet_copy_client.cc:308] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:10:58.857313 2640 tablet_copy_client.cc:523] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:58.858812 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet 23bf7844614647fa86b23155bf69839d from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:58569)
I20240501 20:10:58.858966 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet 23bf7844614647fa86b23155bf69839d from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:58569: session id = cda19d6a81c0473db302535ce2ada7f0-23bf7844614647fa86b23155bf69839d
I20240501 20:10:58.861729 23997 tablet_copy_source_session.cc:205] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:58.866382 2640 tablet_bootstrap.cc:492] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:10:58.870448 2647 tablet_copy_client.cc:308] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.136:44267
I20240501 20:10:58.871176 24130 tablet_copy_service.cc:140] P 8559229e044b4e1ea661b881b3f9af50: Received BeginTabletCopySession request for tablet 2d2e66020ffb438fad69c2ea73762aa7 from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:53433)
I20240501 20:10:58.871315 24130 tablet_copy_service.cc:161] P 8559229e044b4e1ea661b881b3f9af50: Beginning new tablet copy session on tablet 2d2e66020ffb438fad69c2ea73762aa7 from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:53433: session id = cda19d6a81c0473db302535ce2ada7f0-2d2e66020ffb438fad69c2ea73762aa7
I20240501 20:10:58.871606 2642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23bf7844614647fa86b23155bf69839d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.874096 24130 tablet_copy_source_session.cc:205] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:58.877235 2647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2e66020ffb438fad69c2ea73762aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.883648 2647 tablet_copy_client.cc:783] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:58.883983 2647 tablet_copy_client.cc:647] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:58.888310 2642 tablet_copy_client.cc:783] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:58.888612 2642 tablet_copy_client.cc:647] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:58.890765 2647 tablet_copy_client.cc:523] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:58.894942 2647 tablet_bootstrap.cc:492] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:58.895820 2642 tablet_copy_client.cc:523] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:58.903363 2642 tablet_bootstrap.cc:492] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:10:58.935627 2640 tablet_bootstrap.cc:492] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:58.936096 2623 ts_tablet_manager.cc:902] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: Initiating tablet copy from peer 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:10:58.936156 2640 tablet_bootstrap.cc:492] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:10:58.936450 2640 ts_tablet_manager.cc:1367] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.070s user 0.052s sys 0.001s
I20240501 20:10:58.937366 2640 raft_consensus.cc:348] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:58.937647 2640 raft_consensus.cc:729] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.937942 2640 consensus_queue.cc:260] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:10:58.938660 2640 ts_tablet_manager.cc:1398] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:58.939739 20987 tablet_copy_service.cc:342] P cda19d6a81c0473db302535ce2ada7f0: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-75a1b78bb9574b588e6bf69ae7e68bc3 received from {username='slave'} at 127.13.251.136:35157
I20240501 20:10:58.939908 20987 tablet_copy_service.cc:433] P cda19d6a81c0473db302535ce2ada7f0: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-75a1b78bb9574b588e6bf69ae7e68bc3 on tablet 75a1b78bb9574b588e6bf69ae7e68bc3 with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:10:58.964236 2642 tablet_bootstrap.cc:492] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:58.973521 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:58.997911 2647 tablet_bootstrap.cc:492] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:58.998515 2647 tablet_bootstrap.cc:492] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:10:58.998870 2647 ts_tablet_manager.cc:1367] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.104s user 0.021s sys 0.009s
I20240501 20:10:58.999897 2647 raft_consensus.cc:348] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:59.000242 2647 raft_consensus.cc:729] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.000556 2647 consensus_queue.cc:260] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:59.001292 2647 ts_tablet_manager.cc:1398] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:59.005435 24130 tablet_copy_service.cc:342] P 8559229e044b4e1ea661b881b3f9af50: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-2d2e66020ffb438fad69c2ea73762aa7 received from {username='slave'} at 127.13.251.129:53433
I20240501 20:10:59.005563 24130 tablet_copy_service.cc:433] P 8559229e044b4e1ea661b881b3f9af50: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-2d2e66020ffb438fad69c2ea73762aa7 on tablet 2d2e66020ffb438fad69c2ea73762aa7 with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:58.965793 2642 tablet_bootstrap.cc:492] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:10:59.013645 2642 ts_tablet_manager.cc:1367] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.114s user 0.017s sys 0.014s
I20240501 20:10:59.015587 2642 raft_consensus.cc:348] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:59.019349 2642 raft_consensus.cc:729] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.022336 2642 consensus_queue.cc:260] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:10:59.024178 2642 ts_tablet_manager.cc:1398] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 20:10:59.025029 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-23bf7844614647fa86b23155bf69839d received from {username='slave'} at 127.13.251.129:58569
I20240501 20:10:59.025192 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-23bf7844614647fa86b23155bf69839d on tablet 23bf7844614647fa86b23155bf69839d with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:10:59.035672 2623 tablet_copy_client.cc:308] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.135:41019
I20240501 20:10:59.036419 23997 tablet_copy_service.cc:140] P 744c5a532e724391830c0785d32eb2fb: Received BeginTabletCopySession request for tablet 654895cdb221479da79da05bc1938876 from peer feac73db0a3b4aafa374ca5dfd322b08 ({username='slave'} at 127.13.251.130:58437)
I20240501 20:10:59.036562 23997 tablet_copy_service.cc:161] P 744c5a532e724391830c0785d32eb2fb: Beginning new tablet copy session on tablet 654895cdb221479da79da05bc1938876 from peer feac73db0a3b4aafa374ca5dfd322b08 at {username='slave'} at 127.13.251.130:58437: session id = feac73db0a3b4aafa374ca5dfd322b08-654895cdb221479da79da05bc1938876
I20240501 20:10:59.037457 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:59.039479 23997 tablet_copy_source_session.cc:205] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:59.043390 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654895cdb221479da79da05bc1938876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.045542 21098 raft_consensus.cc:1203] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:59.048105 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:59.059967 2623 tablet_copy_client.cc:783] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:59.061326 2623 tablet_copy_client.cc:647] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:59.067390 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:59.141958 2623 tablet_copy_client.cc:523] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:59.150738 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:59.190770 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:59.196624 2623 tablet_bootstrap.cc:492] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap starting.
I20240501 20:10:59.209092 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:59.219174 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:59.230154 2623 tablet_bootstrap.cc:492] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:59.230561 2623 tablet_bootstrap.cc:492] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: Bootstrap complete.
I20240501 20:10:59.230821 2623 ts_tablet_manager.cc:1367] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.006s
I20240501 20:10:59.231669 2623 raft_consensus.cc:348] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:59.231884 2623 raft_consensus.cc:729] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.232118 2623 consensus_queue.cc:260] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:59.232782 2623 ts_tablet_manager.cc:1398] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:59.233678 23997 tablet_copy_service.cc:342] P 744c5a532e724391830c0785d32eb2fb: Request end of tablet copy session feac73db0a3b4aafa374ca5dfd322b08-654895cdb221479da79da05bc1938876 received from {username='slave'} at 127.13.251.130:58437
I20240501 20:10:59.233798 23997 tablet_copy_service.cc:433] P 744c5a532e724391830c0785d32eb2fb: ending tablet copy session feac73db0a3b4aafa374ca5dfd322b08-654895cdb221479da79da05bc1938876 on tablet 654895cdb221479da79da05bc1938876 with peer feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:10:59.241531 24107 raft_consensus.cc:1203] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:59.295459 20963 raft_consensus.cc:1203] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:59.344985 20963 raft_consensus.cc:1203] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:59.438766 21098 raft_consensus.cc:1203] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:59.699038 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:59.705731 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:59.714046 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:10:59.721181 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:59.782824 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:59.785658 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:59.801784 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:59.853333 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:59.945631 23975 consensus_queue.cc:237] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:10:59.948065 21356 raft_consensus.cc:1261] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:59.948421 21098 raft_consensus.cc:1261] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:59.949421 2735 consensus_queue.cc:1035] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:59.949925 2735 consensus_queue.cc:1035] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:59.961009 2359 raft_consensus.cc:2932] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:59.961814 21356 raft_consensus.cc:2932] T 654895cdb221479da79da05bc1938876 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:59.962008 21096 raft_consensus.cc:2932] T 654895cdb221479da79da05bc1938876 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:59.968899 20822 catalog_manager.cc:5484] T 654895cdb221479da79da05bc1938876 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:10:59.976167 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 654895cdb221479da79da05bc1938876 with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:10:59.996734 2777 tablet_replica.cc:330] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:10:59.996937 2777 raft_consensus.cc:2229] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:59.997169 2777 raft_consensus.cc:2258] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:59.997756 2777 ts_tablet_manager.cc:1875] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:00.010206 2777 ts_tablet_manager.cc:1888] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:00.010448 2777 log.cc:1196] T 654895cdb221479da79da05bc1938876 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/654895cdb221479da79da05bc1938876
I20240501 20:11:00.011561 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 654895cdb221479da79da05bc1938876 (table rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]) successfully deleted
I20240501 20:11:00.343837 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:00.381587 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:00.389858 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:00.439445 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:00.466229 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:11:00.471408 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:00.477708 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:00.569494 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:00.732738 20965 consensus_queue.cc:237] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:00.734772 24107 raft_consensus.cc:1261] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:00.734793 21096 raft_consensus.cc:1261] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:00.735275 2837 consensus_queue.cc:1035] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:00.735522 2342 consensus_queue.cc:1035] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:00.739459 2838 raft_consensus.cc:2932] T 75a1b78bb9574b588e6bf69ae7e68bc3 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:00.740131 24107 raft_consensus.cc:2932] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:00.740602 21096 raft_consensus.cc:2932] T 75a1b78bb9574b588e6bf69ae7e68bc3 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:00.746820 20825 catalog_manager.cc:5484] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:00.759111 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a1b78bb9574b588e6bf69ae7e68bc3 with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42940
I20240501 20:11:00.759841 2862 tablet_replica.cc:330] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:11:00.760176 2862 raft_consensus.cc:2229] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:00.760673 2862 raft_consensus.cc:2258] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:00.762089 2862 ts_tablet_manager.cc:1875] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:00.770509 2862 ts_tablet_manager.cc:1888] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:11:00.770764 2862 log.cc:1196] T 75a1b78bb9574b588e6bf69ae7e68bc3 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/75a1b78bb9574b588e6bf69ae7e68bc3
I20240501 20:11:00.771760 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 75a1b78bb9574b588e6bf69ae7e68bc3 (table rebalance_test_table_19 [id=aa669f7164f34cfba57f458802ccd128]) successfully deleted
I20240501 20:11:01.214038 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:01.233866 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:01.241238 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:01.284579 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:11:01.313239 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:01.323699 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:01.324321 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:01.334115 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:01.600498 23977 consensus_queue.cc:237] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:01.602607 20963 raft_consensus.cc:1261] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:01.602689 21098 raft_consensus.cc:1261] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 744c5a532e724391830c0785d32eb2fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:01.603129 2359 consensus_queue.cc:1035] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:01.603619 2359 consensus_queue.cc:1035] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:01.605819 2836 raft_consensus.cc:2932] T 23bf7844614647fa86b23155bf69839d P 744c5a532e724391830c0785d32eb2fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:01.606444 20963 raft_consensus.cc:2932] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:01.614881 20818 catalog_manager.cc:5484] T 23bf7844614647fa86b23155bf69839d P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "744c5a532e724391830c0785d32eb2fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:01.619784 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 23bf7844614647fa86b23155bf69839d with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:11:01.620358 2948 tablet_replica.cc:330] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:11:01.620558 2948 raft_consensus.cc:2229] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:01.620759 2948 raft_consensus.cc:2258] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:01.621290 2948 ts_tablet_manager.cc:1875] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:01.624310 21098 raft_consensus.cc:2932] T 23bf7844614647fa86b23155bf69839d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:01.634573 2948 ts_tablet_manager.cc:1888] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:01.634689 2948 log.cc:1196] T 23bf7844614647fa86b23155bf69839d P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/23bf7844614647fa86b23155bf69839d
I20240501 20:11:01.635362 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 23bf7844614647fa86b23155bf69839d (table rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]) successfully deleted
I20240501 20:11:01.892488 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:01.893460 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:01.919226 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:01.949095 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:01.952023 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:01.952543 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:01.959713 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 51629.7us
I20240501 20:11:01.973964 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:01.976590 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:11:02.180683 23846 consensus_queue.cc:237] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } }
I20240501 20:11:02.182420 21098 raft_consensus.cc:1261] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:02.182687 20963 raft_consensus.cc:1261] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:02.182924 2681 consensus_queue.cc:1035] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:02.183351 2681 consensus_queue.cc:1035] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:02.185304 2681 raft_consensus.cc:2932] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:11:02.188239 21098 raft_consensus.cc:2932] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:11:02.188691 20963 raft_consensus.cc:2932] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:11:02.193010 20824 catalog_manager.cc:5484] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 2 to 3, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } }
I20240501 20:11:02.203419 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f23f1dafeb4447b33cb7f5bdbf03c4 with delete_type TABLET_DATA_TOMBSTONED (TS 51bf640e895b4e5eb91bdf7509ea7e46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47478
I20240501 20:11:02.218152 3037 tablet_replica.cc:330] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46: stopping tablet replica
I20240501 20:11:02.218395 3037 raft_consensus.cc:2229] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:02.218591 3037 raft_consensus.cc:2258] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:02.219200 3037 ts_tablet_manager.cc:1875] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:02.225531 3037 ts_tablet_manager.cc:1888] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:02.225662 3037 log.cc:1196] T 74f23f1dafeb4447b33cb7f5bdbf03c4 P 51bf640e895b4e5eb91bdf7509ea7e46: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-4/wal/wals/74f23f1dafeb4447b33cb7f5bdbf03c4
I20240501 20:11:02.226433 20812 catalog_manager.cc:4830] TS 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): tablet 74f23f1dafeb4447b33cb7f5bdbf03c4 (table rebalancer_extra_table_20 [id=acf2d948308a4361b17a298ebd29a7e0]) successfully deleted
I20240501 20:11:02.519865 21338 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 20:11:02.535600 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:02.564343 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:02.582967 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:02.598050 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:02.630609 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:02.633029 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:02.664352 21077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:02.818166 21357 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998f71aeae984fd5b866cb866bd4e87a"
dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37544
I20240501 20:11:02.818326 21357 raft_consensus.cc:595] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:02.824067 2922 raft_consensus.cc:979] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: : Instructing follower feac73db0a3b4aafa374ca5dfd322b08 to start an election
I20240501 20:11:02.824220 2922 raft_consensus.cc:1067] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Signalling peer feac73db0a3b4aafa374ca5dfd322b08 to start an election
I20240501 20:11:02.824811 21098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "998f71aeae984fd5b866cb866bd4e87a"
dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
from {username='slave'} at 127.13.251.132:47489
I20240501 20:11:02.824992 21098 raft_consensus.cc:482] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:02.825107 21098 raft_consensus.cc:3037] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:02.827286 21098 raft_consensus.cc:504] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:02.828996 20963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998f71aeae984fd5b866cb866bd4e87a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:11:02.828960 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998f71aeae984fd5b866cb866bd4e87a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:11:02.829176 20963 raft_consensus.cc:3037] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:02.829176 21357 raft_consensus.cc:3032] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:02.829242 21357 raft_consensus.cc:729] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:11:02.829481 21357 consensus_queue.cc:260] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:02.829845 21357 raft_consensus.cc:3037] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:02.831754 20963 raft_consensus.cc:2454] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:11:02.832401 21357 raft_consensus.cc:2454] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:11:02.832782 21032 leader_election.cc:302] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, cda19d6a81c0473db302535ce2ada7f0, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:11:02.833070 2101 raft_consensus.cc:2781] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:02.833195 2101 raft_consensus.cc:686] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 2 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:11:02.833487 2101 consensus_queue.cc:237] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:02.834740 21098 leader_election.cc:288] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:02.835276 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998f71aeae984fd5b866cb866bd4e87a" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:11:02.835474 23846 raft_consensus.cc:3037] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:02.837888 23846 raft_consensus.cc:2454] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:11:02.838443 20825 catalog_manager.cc:5484] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 1 to 2, leader changed from 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 2 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:03.113476 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:03.120987 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:03.162504 21338 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 20:11:03.233999 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:03.252115 20963 raft_consensus.cc:1261] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:11:03.252779 2101 consensus_queue.cc:1035] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:03.267637 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:03.275635 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:03.307389 23846 raft_consensus.cc:1261] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:11:03.311776 32014 consensus_queue.cc:1035] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:03.329267 21358 raft_consensus.cc:1261] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:11:03.330215 2101 consensus_queue.cc:1035] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:03.345117 21077 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:03.378159 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:03.755977 23842 consensus_queue.cc:237] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:03.758524 23709 raft_consensus.cc:1261] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:03.758666 23977 raft_consensus.cc:1261] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:03.759243 2681 consensus_queue.cc:1035] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:11:03.759860 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf1ac1f2f3d642e9a9c785bb4963af1f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:03.759896 3031 consensus_queue.cc:1035] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:03.776232 3031 raft_consensus.cc:2932] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:03.776893 23977 raft_consensus.cc:2932] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:03.785881 23709 raft_consensus.cc:2932] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:03.786293 20825 catalog_manager.cc:5484] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:03.823750 21358 consensus_queue.cc:237] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:03.826680 21228 raft_consensus.cc:1261] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:03.826761 21293 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b70e144824074927bb0f4eafcdeadcec. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:03.827323 3150 consensus_queue.cc:1035] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:03.827519 21098 raft_consensus.cc:1261] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:03.828414 2922 consensus_queue.cc:1035] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:03.844306 3150 raft_consensus.cc:2932] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:03.845933 21098 raft_consensus.cc:2932] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:03.849190 21228 raft_consensus.cc:2932] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:03.851094 20823 catalog_manager.cc:5484] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:03.913607 23842 consensus_queue.cc:237] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:03.915891 24109 raft_consensus.cc:1261] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:11:03.916376 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 224f83ad3d514a7fa5cdfa4e70ff2fd8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:03.916477 23977 raft_consensus.cc:1261] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:03.916632 2681 consensus_queue.cc:1035] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:03.916962 3031 consensus_queue.cc:1035] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:03.919643 2681 raft_consensus.cc:2932] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:03.920236 23977 raft_consensus.cc:2932] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:03.923249 24109 raft_consensus.cc:2932] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:03.927466 20821 catalog_manager.cc:5484] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:03.974922 21357 consensus_queue.cc:237] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:03.977312 21098 raft_consensus.cc:1261] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:03.977916 3150 consensus_queue.cc:1035] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:03.980053 20965 raft_consensus.cc:1261] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:03.980798 21293 consensus_peers.cc:485] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca03186ff8b342dd958b205125b14177. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:03.980938 3150 consensus_queue.cc:1035] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:03.983726 3150 raft_consensus.cc:2932] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:03.987825 20965 raft_consensus.cc:2932] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:03.995986 21098 raft_consensus.cc:2932] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.000797 20821 catalog_manager.cc:5484] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.057480 23711 consensus_queue.cc:237] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:04.060077 24109 raft_consensus.cc:1261] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:11:04.060406 23644 consensus_peers.cc:485] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259): Couldn't send request to peer cda19d6a81c0473db302535ce2ada7f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 024bb310955b4a2e80d908b8c0a8e4c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:04.060518 23842 raft_consensus.cc:1261] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:04.060595 3286 consensus_queue.cc:1035] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:04.061107 3286 consensus_queue.cc:1035] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:04.072926 3286 raft_consensus.cc:2932] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:04.073415 24109 raft_consensus.cc:2932] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:04.077404 23842 raft_consensus.cc:2932] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:04.080390 20821 catalog_manager.cc:5484] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: config changed from index -1 to 2, VOTER cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129) added. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:04.144559 21098 consensus_queue.cc:237] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:04.148097 21228 raft_consensus.cc:1261] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:04.148077 21357 raft_consensus.cc:1261] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:04.148670 2101 consensus_queue.cc:1035] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:04.148962 32014 consensus_queue.cc:1035] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:04.150432 21033 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267): Couldn't send request to peer 8559229e044b4e1ea661b881b3f9af50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1476e8f4c9343928f68365fd2772f0f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:04.156620 2101 raft_consensus.cc:2932] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:04.157248 21358 raft_consensus.cc:2932] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:04.161906 21228 raft_consensus.cc:2932] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:04.162923 20825 catalog_manager.cc:5484] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:04.207784 24109 consensus_queue.cc:237] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
W20240501 20:11:04.210482 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4341f17784bc45359a9648ab47c39628. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:04.210660 23977 raft_consensus.cc:1261] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:04.210672 23842 raft_consensus.cc:1261] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:04.211145 1855 consensus_queue.cc:1035] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:04.211419 3159 consensus_queue.cc:1035] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:04.215247 1855 raft_consensus.cc:2932] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:04.216244 23842 raft_consensus.cc:2932] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:04.220341 23977 raft_consensus.cc:2932] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:04.223582 20825 catalog_manager.cc:5484] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:04.248826 3390 ts_tablet_manager.cc:902] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:04.250217 3390 tablet_copy_client.cc:308] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:11:04.251063 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet cf1ac1f2f3d642e9a9c785bb4963af1f from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:54857)
I20240501 20:11:04.251216 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet cf1ac1f2f3d642e9a9c785bb4963af1f from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:54857: session id = 4af096f5a6a9440a94f6682b399efe20-cf1ac1f2f3d642e9a9c785bb4963af1f
I20240501 20:11:04.253800 23866 tablet_copy_source_session.cc:205] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:04.255455 3390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1ac1f2f3d642e9a9c785bb4963af1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.261222 3390 tablet_copy_client.cc:783] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:04.261547 3390 tablet_copy_client.cc:647] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:04.263252 3390 tablet_copy_client.cc:523] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:04.265866 3390 tablet_bootstrap.cc:492] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:11:04.272449 3390 tablet_bootstrap.cc:492] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:04.272827 3390 tablet_bootstrap.cc:492] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:11:04.273059 3390 ts_tablet_manager.cc:1367] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.003s
I20240501 20:11:04.273887 3390 raft_consensus.cc:348] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:04.274132 3390 raft_consensus.cc:729] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:11:04.274339 3390 consensus_queue.cc:260] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:04.275249 3390 ts_tablet_manager.cc:1398] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:11:04.275970 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-cf1ac1f2f3d642e9a9c785bb4963af1f received from {username='slave'} at 127.13.251.131:54857
I20240501 20:11:04.276101 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-cf1ac1f2f3d642e9a9c785bb4963af1f on tablet cf1ac1f2f3d642e9a9c785bb4963af1f with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:04.293084 21358 consensus_queue.cc:237] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:04.298626 21098 raft_consensus.cc:1261] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:04.299470 21228 raft_consensus.cc:1261] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:04.299887 21293 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4c4664b17f04e2d8285b7851b27d1a1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:04.300056 3283 consensus_queue.cc:1035] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:04.300328 3150 consensus_queue.cc:1035] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:04.304764 3283 raft_consensus.cc:2932] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.309231 21228 raft_consensus.cc:2932] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.311873 20821 catalog_manager.cc:5484] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.314791 21098 raft_consensus.cc:2932] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.356843 3390 ts_tablet_manager.cc:902] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:04.360544 3390 tablet_copy_client.cc:308] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.134:42431
I20240501 20:11:04.361158 23866 tablet_copy_service.cc:140] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Received BeginTabletCopySession request for tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8 from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:54857)
I20240501 20:11:04.361303 23866 tablet_copy_service.cc:161] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Beginning new tablet copy session on tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8 from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:54857: session id = 4af096f5a6a9440a94f6682b399efe20-224f83ad3d514a7fa5cdfa4e70ff2fd8
I20240501 20:11:04.363948 23866 tablet_copy_source_session.cc:205] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:04.365083 3390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.370196 3390 tablet_copy_client.cc:783] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:04.370455 3390 tablet_copy_client.cc:647] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:04.371956 3390 tablet_copy_client.cc:523] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:04.377684 3390 tablet_bootstrap.cc:492] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:11:04.383975 3390 tablet_bootstrap.cc:492] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:04.384294 3390 tablet_bootstrap.cc:492] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:11:04.384506 3390 ts_tablet_manager.cc:1367] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240501 20:11:04.385578 3390 raft_consensus.cc:348] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:04.385757 3390 raft_consensus.cc:729] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:11:04.385959 3390 consensus_queue.cc:260] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:04.386502 3390 ts_tablet_manager.cc:1398] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:04.388095 23866 tablet_copy_service.cc:342] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-224f83ad3d514a7fa5cdfa4e70ff2fd8 received from {username='slave'} at 127.13.251.131:54857
I20240501 20:11:04.388192 23866 tablet_copy_service.cc:433] P c6dcaa8efff642dcb7c2c6e8be3a65f7: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-224f83ad3d514a7fa5cdfa4e70ff2fd8 on tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8 with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:04.397130 21358 consensus_queue.cc:237] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:04.399367 23842 raft_consensus.cc:1261] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:04.399787 23711 raft_consensus.cc:1261] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:11:04.399886 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 236ba60733e1416395346179be3b3888. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:04.400175 3150 consensus_queue.cc:1035] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:04.400772 3283 consensus_queue.cc:1035] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:04.403934 2922 raft_consensus.cc:2932] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:04.404574 23711 raft_consensus.cc:2932] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:04.409950 23842 raft_consensus.cc:2932] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:04.410710 20821 catalog_manager.cc:5484] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:04.420753 3429 ts_tablet_manager.cc:902] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:11:04.430137 3429 tablet_copy_client.cc:308] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:11:04.430967 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet b70e144824074927bb0f4eafcdeadcec from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:58619)
I20240501 20:11:04.431174 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet b70e144824074927bb0f4eafcdeadcec from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:58619: session id = 8559229e044b4e1ea661b881b3f9af50-b70e144824074927bb0f4eafcdeadcec
I20240501 20:11:04.435245 21378 tablet_copy_source_session.cc:205] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:04.438944 3429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70e144824074927bb0f4eafcdeadcec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.444900 3429 tablet_copy_client.cc:783] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:04.445096 3429 tablet_copy_client.cc:647] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:04.447763 3429 tablet_copy_client.cc:523] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:04.452720 3429 tablet_bootstrap.cc:492] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:11:04.475739 21228 consensus_queue.cc:237] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:04.485743 21358 raft_consensus.cc:1261] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:04.487195 2993 consensus_queue.cc:1035] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:04.489193 21163 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24cc6ff1a0104ac690c8d8f82c17200c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:04.496376 21098 raft_consensus.cc:1261] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:04.497416 2993 consensus_queue.cc:1035] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:04.520918 3429 tablet_bootstrap.cc:492] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:04.521378 3429 tablet_bootstrap.cc:492] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:11:04.521603 3429 ts_tablet_manager.cc:1367] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.069s user 0.046s sys 0.000s
I20240501 20:11:04.522156 3429 raft_consensus.cc:348] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:04.522318 3429 raft_consensus.cc:729] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:11:04.522523 3429 consensus_queue.cc:260] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:04.523038 3429 ts_tablet_manager.cc:1398] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:11:04.528570 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-b70e144824074927bb0f4eafcdeadcec received from {username='slave'} at 127.13.251.136:58619
I20240501 20:11:04.528702 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-b70e144824074927bb0f4eafcdeadcec on tablet b70e144824074927bb0f4eafcdeadcec with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:11:04.530583 3454 raft_consensus.cc:2932] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.531283 21098 raft_consensus.cc:2932] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.540728 20823 catalog_manager.cc:5484] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.563347 21358 raft_consensus.cc:2932] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:04.594137 3459 ts_tablet_manager.cc:902] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:11:04.604646 3390 ts_tablet_manager.cc:902] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:11:04.644546 3474 ts_tablet_manager.cc:902] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: Initiating tablet copy from peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:11:04.648232 3390 tablet_copy_client.cc:308] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.136:44267
I20240501 20:11:04.648855 24130 tablet_copy_service.cc:140] P 8559229e044b4e1ea661b881b3f9af50: Received BeginTabletCopySession request for tablet 4341f17784bc45359a9648ab47c39628 from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:53727)
I20240501 20:11:04.649040 24130 tablet_copy_service.cc:161] P 8559229e044b4e1ea661b881b3f9af50: Beginning new tablet copy session on tablet 4341f17784bc45359a9648ab47c39628 from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:53727: session id = 4af096f5a6a9440a94f6682b399efe20-4341f17784bc45359a9648ab47c39628
I20240501 20:11:04.649180 3459 tablet_copy_client.cc:308] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:11:04.650171 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet ca03186ff8b342dd958b205125b14177 from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:37097)
I20240501 20:11:04.650334 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet ca03186ff8b342dd958b205125b14177 from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:37097: session id = 51bf640e895b4e5eb91bdf7509ea7e46-ca03186ff8b342dd958b205125b14177
I20240501 20:11:04.653193 21378 tablet_copy_source_session.cc:205] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:04.653699 24130 tablet_copy_source_session.cc:205] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:04.655455 3390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4341f17784bc45359a9648ab47c39628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.655580 3459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca03186ff8b342dd958b205125b14177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.662358 21228 raft_consensus.cc:1203] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:11:04.663829 3390 tablet_copy_client.cc:783] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:04.664047 3390 tablet_copy_client.cc:647] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:04.668831 3390 tablet_copy_client.cc:523] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:04.673343 3390 tablet_bootstrap.cc:492] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:11:04.674153 3459 tablet_copy_client.cc:783] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:04.674396 3459 tablet_copy_client.cc:647] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:04.680605 3459 tablet_copy_client.cc:523] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:04.684127 3459 tablet_bootstrap.cc:492] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:11:04.687906 3474 tablet_copy_client.cc:308] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.133:37155
I20240501 20:11:04.688706 23734 tablet_copy_service.cc:140] P 51bf640e895b4e5eb91bdf7509ea7e46: Received BeginTabletCopySession request for tablet 024bb310955b4a2e80d908b8c0a8e4c1 from peer cda19d6a81c0473db302535ce2ada7f0 ({username='slave'} at 127.13.251.129:59353)
I20240501 20:11:04.688848 23734 tablet_copy_service.cc:161] P 51bf640e895b4e5eb91bdf7509ea7e46: Beginning new tablet copy session on tablet 024bb310955b4a2e80d908b8c0a8e4c1 from peer cda19d6a81c0473db302535ce2ada7f0 at {username='slave'} at 127.13.251.129:59353: session id = cda19d6a81c0473db302535ce2ada7f0-024bb310955b4a2e80d908b8c0a8e4c1
I20240501 20:11:04.690433 3390 tablet_bootstrap.cc:492] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:04.690796 3390 tablet_bootstrap.cc:492] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:11:04.691087 3390 ts_tablet_manager.cc:1367] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.001s
I20240501 20:11:04.691556 23734 tablet_copy_source_session.cc:205] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:04.691851 3390 raft_consensus.cc:348] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:04.692132 3390 raft_consensus.cc:729] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:11:04.692914 3474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024bb310955b4a2e80d908b8c0a8e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.699719 3390 consensus_queue.cc:260] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:04.700471 3390 ts_tablet_manager.cc:1398] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:11:04.702069 24130 tablet_copy_service.cc:342] P 8559229e044b4e1ea661b881b3f9af50: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-4341f17784bc45359a9648ab47c39628 received from {username='slave'} at 127.13.251.131:53727
I20240501 20:11:04.702204 24130 tablet_copy_service.cc:433] P 8559229e044b4e1ea661b881b3f9af50: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-4341f17784bc45359a9648ab47c39628 on tablet 4341f17784bc45359a9648ab47c39628 with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:04.703248 3474 tablet_copy_client.cc:783] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:04.703509 3474 tablet_copy_client.cc:647] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:04.705485 3474 tablet_copy_client.cc:523] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:04.711553 3474 tablet_bootstrap.cc:492] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap starting.
I20240501 20:11:04.717959 3474 tablet_bootstrap.cc:492] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:04.718355 3474 tablet_bootstrap.cc:492] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: Bootstrap complete.
I20240501 20:11:04.718712 3474 ts_tablet_manager.cc:1367] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240501 20:11:04.719378 3474 raft_consensus.cc:348] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:04.719637 3474 raft_consensus.cc:729] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:04.719895 3474 consensus_queue.cc:260] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:04.720536 3474 ts_tablet_manager.cc:1398] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:04.721143 23734 tablet_copy_service.cc:342] P 51bf640e895b4e5eb91bdf7509ea7e46: Request end of tablet copy session cda19d6a81c0473db302535ce2ada7f0-024bb310955b4a2e80d908b8c0a8e4c1 received from {username='slave'} at 127.13.251.129:59353
I20240501 20:11:04.721238 23734 tablet_copy_service.cc:433] P 51bf640e895b4e5eb91bdf7509ea7e46: ending tablet copy session cda19d6a81c0473db302535ce2ada7f0-024bb310955b4a2e80d908b8c0a8e4c1 on tablet 024bb310955b4a2e80d908b8c0a8e4c1 with peer cda19d6a81c0473db302535ce2ada7f0
I20240501 20:11:04.758569 3429 ts_tablet_manager.cc:902] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:11:04.775945 3459 tablet_bootstrap.cc:492] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:04.776350 3459 tablet_bootstrap.cc:492] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:11:04.776605 3459 ts_tablet_manager.cc:1367] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.093s user 0.057s sys 0.008s
I20240501 20:11:04.777377 3459 raft_consensus.cc:348] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:04.777688 3459 raft_consensus.cc:729] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:11:04.777959 3459 consensus_queue.cc:260] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:04.778666 3459 ts_tablet_manager.cc:1398] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:11:04.779712 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-ca03186ff8b342dd958b205125b14177 received from {username='slave'} at 127.13.251.133:37097
I20240501 20:11:04.779847 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-ca03186ff8b342dd958b205125b14177 on tablet ca03186ff8b342dd958b205125b14177 with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:11:04.784622 3429 tablet_copy_client.cc:308] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:11:04.785648 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet e1476e8f4c9343928f68365fd2772f0f from peer 8559229e044b4e1ea661b881b3f9af50 ({username='slave'} at 127.13.251.136:53785)
I20240501 20:11:04.785815 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet e1476e8f4c9343928f68365fd2772f0f from peer 8559229e044b4e1ea661b881b3f9af50 at {username='slave'} at 127.13.251.136:53785: session id = 8559229e044b4e1ea661b881b3f9af50-e1476e8f4c9343928f68365fd2772f0f
I20240501 20:11:04.789517 21118 tablet_copy_source_session.cc:205] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:04.790910 3429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1476e8f4c9343928f68365fd2772f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.799093 3429 tablet_copy_client.cc:783] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:04.799445 3429 tablet_copy_client.cc:647] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:04.805083 3429 tablet_copy_client.cc:523] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:04.809967 3429 tablet_bootstrap.cc:492] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: Bootstrap starting.
I20240501 20:11:04.841655 3459 ts_tablet_manager.cc:902] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:11:04.865118 3459 tablet_copy_client.cc:308] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:11:04.865856 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet b4c4664b17f04e2d8285b7851b27d1a1 from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:37097)
I20240501 20:11:04.866034 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet b4c4664b17f04e2d8285b7851b27d1a1 from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:37097: session id = 51bf640e895b4e5eb91bdf7509ea7e46-b4c4664b17f04e2d8285b7851b27d1a1
I20240501 20:11:04.868521 21378 tablet_copy_source_session.cc:205] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:04.870342 3459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c4664b17f04e2d8285b7851b27d1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.877125 3459 tablet_copy_client.cc:783] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:04.877594 3459 tablet_copy_client.cc:647] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:04.885172 3459 tablet_copy_client.cc:523] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:04.888664 3459 tablet_bootstrap.cc:492] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:11:04.896256 24107 raft_consensus.cc:1203] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:04.910131 21228 raft_consensus.cc:1203] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:11:04.931708 3429 tablet_bootstrap.cc:492] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:04.932338 3429 tablet_bootstrap.cc:492] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: Bootstrap complete.
I20240501 20:11:04.932686 3429 ts_tablet_manager.cc:1367] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: Time spent bootstrapping tablet: real 0.123s user 0.073s sys 0.010s
I20240501 20:11:04.933480 3429 raft_consensus.cc:348] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:04.933808 3429 raft_consensus.cc:729] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Initialized, Role: FOLLOWER
I20240501 20:11:04.934078 3429 consensus_queue.cc:260] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:04.934850 3429 ts_tablet_manager.cc:1398] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:11:04.935994 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session 8559229e044b4e1ea661b881b3f9af50-e1476e8f4c9343928f68365fd2772f0f received from {username='slave'} at 127.13.251.136:53785
I20240501 20:11:04.936144 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session 8559229e044b4e1ea661b881b3f9af50-e1476e8f4c9343928f68365fd2772f0f on tablet e1476e8f4c9343928f68365fd2772f0f with peer 8559229e044b4e1ea661b881b3f9af50
I20240501 20:11:04.939159 3390 ts_tablet_manager.cc:902] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:11:05.024745 3525 ts_tablet_manager.cc:902] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:05.084371 20963 raft_consensus.cc:1203] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:11:05.119112 3459 tablet_bootstrap.cc:492] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:05.120095 3459 tablet_bootstrap.cc:492] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:11:05.120359 3459 ts_tablet_manager.cc:1367] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.232s user 0.066s sys 0.004s
I20240501 20:11:05.121006 3459 raft_consensus.cc:348] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:05.121224 3459 raft_consensus.cc:729] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:11:05.124186 3459 consensus_queue.cc:260] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:05.124856 3459 ts_tablet_manager.cc:1398] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:11:05.127307 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-b4c4664b17f04e2d8285b7851b27d1a1 received from {username='slave'} at 127.13.251.133:37097
I20240501 20:11:05.127439 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-b4c4664b17f04e2d8285b7851b27d1a1 on tablet b4c4664b17f04e2d8285b7851b27d1a1 with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:11:05.133822 3525 tablet_copy_client.cc:308] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.131:38349
I20240501 20:11:05.134778 21248 tablet_copy_service.cc:140] P 4af096f5a6a9440a94f6682b399efe20: Received BeginTabletCopySession request for tablet 24cc6ff1a0104ac690c8d8f82c17200c from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:38225)
I20240501 20:11:05.134940 21248 tablet_copy_service.cc:161] P 4af096f5a6a9440a94f6682b399efe20: Beginning new tablet copy session on tablet 24cc6ff1a0104ac690c8d8f82c17200c from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:38225: session id = 51bf640e895b4e5eb91bdf7509ea7e46-24cc6ff1a0104ac690c8d8f82c17200c
I20240501 20:11:05.142680 21248 tablet_copy_source_session.cc:205] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:05.152431 3390 tablet_copy_client.cc:308] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:11:05.155843 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet 236ba60733e1416395346179be3b3888 from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:46863)
I20240501 20:11:05.156014 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet 236ba60733e1416395346179be3b3888 from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:46863: session id = 4af096f5a6a9440a94f6682b399efe20-236ba60733e1416395346179be3b3888
I20240501 20:11:05.158984 21378 tablet_copy_source_session.cc:205] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:05.167444 3390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236ba60733e1416395346179be3b3888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:05.173656 3390 tablet_copy_client.cc:783] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:05.173916 3390 tablet_copy_client.cc:647] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:05.182180 3525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc6ff1a0104ac690c8d8f82c17200c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:05.187633 3390 tablet_copy_client.cc:523] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:05.191159 3390 tablet_bootstrap.cc:492] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
I20240501 20:11:05.192250 23711 raft_consensus.cc:1203] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:05.196449 3525 tablet_copy_client.cc:783] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:05.196661 3525 tablet_copy_client.cc:647] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:05.200875 3525 tablet_copy_client.cc:523] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:05.214450 3525 tablet_bootstrap.cc:492] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:11:05.232021 21228 raft_consensus.cc:1203] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:11:05.290863 23711 raft_consensus.cc:1203] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:05.375411 24108 raft_consensus.cc:1203] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:05.388068 3525 tablet_bootstrap.cc:492] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:05.388646 3525 tablet_bootstrap.cc:492] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:11:05.388967 3525 ts_tablet_manager.cc:1367] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.175s user 0.065s sys 0.010s
I20240501 20:11:05.389954 3525 raft_consensus.cc:348] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:05.397851 3525 raft_consensus.cc:729] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:11:05.398357 3525 consensus_queue.cc:260] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:05.399392 3525 ts_tablet_manager.cc:1398] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.010s user 0.000s sys 0.004s
I20240501 20:11:05.402014 21248 tablet_copy_service.cc:342] P 4af096f5a6a9440a94f6682b399efe20: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-24cc6ff1a0104ac690c8d8f82c17200c received from {username='slave'} at 127.13.251.133:38225
I20240501 20:11:05.402158 21248 tablet_copy_service.cc:433] P 4af096f5a6a9440a94f6682b399efe20: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-24cc6ff1a0104ac690c8d8f82c17200c on tablet 24cc6ff1a0104ac690c8d8f82c17200c with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:11:05.520612 23709 raft_consensus.cc:1203] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:05.557355 3390 tablet_bootstrap.cc:492] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:05.557813 3390 tablet_bootstrap.cc:492] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: Bootstrap complete.
I20240501 20:11:05.558077 3390 ts_tablet_manager.cc:1367] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: Time spent bootstrapping tablet: real 0.367s user 0.135s sys 0.166s
I20240501 20:11:05.558871 3390 raft_consensus.cc:348] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:05.559165 3390 raft_consensus.cc:729] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af096f5a6a9440a94f6682b399efe20, State: Initialized, Role: FOLLOWER
I20240501 20:11:05.562852 3390 consensus_queue.cc:260] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:05.564937 3390 ts_tablet_manager.cc:1398] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 20:11:05.565560 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 4af096f5a6a9440a94f6682b399efe20-236ba60733e1416395346179be3b3888 received from {username='slave'} at 127.13.251.131:46863
I20240501 20:11:05.565707 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 4af096f5a6a9440a94f6682b399efe20-236ba60733e1416395346179be3b3888 on tablet 236ba60733e1416395346179be3b3888 with peer 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:05.747653 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:05.769652 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:05.770187 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:05.774590 21338 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 20:11:05.817106 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:05.852716 21228 raft_consensus.cc:1203] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:11:05.897517 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:05.915973 21077 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:05.930476 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:06.342504 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:06.348269 21338 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 20:11:06.356489 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:06.399410 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:06.439808 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:06.440194 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:06.456249 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:06.511104 21077 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:06.754180 21357 consensus_queue.cc:237] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:06.756403 21227 raft_consensus.cc:1261] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:06.757015 23709 raft_consensus.cc:1261] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:06.756982 3561 consensus_queue.cc:1035] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:06.757633 3561 consensus_queue.cc:1035] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:06.759861 3283 raft_consensus.cc:2932] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:06.761042 21227 raft_consensus.cc:2932] T b4c4664b17f04e2d8285b7851b27d1a1 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:06.763890 23711 raft_consensus.cc:2932] T b4c4664b17f04e2d8285b7851b27d1a1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:06.767489 20824 catalog_manager.cc:5484] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:06.775993 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet b4c4664b17f04e2d8285b7851b27d1a1 with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51330
I20240501 20:11:06.779048 3655 tablet_replica.cc:330] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:11:06.779764 3655 raft_consensus.cc:2229] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:06.780133 3655 raft_consensus.cc:2258] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:06.780767 3655 ts_tablet_manager.cc:1875] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:06.788733 3655 ts_tablet_manager.cc:1888] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:11:06.788887 3655 log.cc:1196] T b4c4664b17f04e2d8285b7851b27d1a1 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/b4c4664b17f04e2d8285b7851b27d1a1
I20240501 20:11:06.789692 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet b4c4664b17f04e2d8285b7851b27d1a1 (table rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]) successfully deleted
I20240501 20:11:07.079752 21338 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 20:11:07.080194 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:07.106076 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:07.110801 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:07.180977 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(f392e77df7ff4da489b4d8a8dc415af0): perf score=0.033359
I20240501 20:11:07.209698 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:07.221469 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(f392e77df7ff4da489b4d8a8dc415af0): perf score=0.033374
I20240501 20:11:07.229169 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(420b46edf5fc438fb01e822440bce8f1): perf score=0.033381
I20240501 20:11:07.250495 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:07.258234 21077 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:07.267720 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:07.287212 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(420b46edf5fc438fb01e822440bce8f1): perf score=0.033400
I20240501 20:11:07.657815 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(420b46edf5fc438fb01e822440bce8f1) complete. Timing: real 0.428s user 0.079s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":58676,"dirs.queue_time_us":45304,"dirs.run_cpu_time_us":361,"dirs.run_wall_time_us":3389,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9425,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":152107,"threads_started":3,"wal-append.queue_time_us":42564}
I20240501 20:11:07.666911 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(420b46edf5fc438fb01e822440bce8f1): 4921 bytes on disk
I20240501 20:11:07.669647 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(420b46edf5fc438fb01e822440bce8f1) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 20:11:07.672469 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(24cc6ff1a0104ac690c8d8f82c17200c): perf score=0.033501
W20240501 20:11:07.804546 21161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 110626us
I20240501 20:11:07.869015 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(f392e77df7ff4da489b4d8a8dc415af0) complete. Timing: real 0.680s user 0.064s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":451,"dirs.queue_time_us":5922,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":2455,"drs_written":1,"lbm_read_time_us":160,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1088,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":62087,"threads_started":3,"wal-append.queue_time_us":21345}
I20240501 20:11:07.880600 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(f392e77df7ff4da489b4d8a8dc415af0): 4834 bytes on disk
I20240501 20:11:07.891902 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(f392e77df7ff4da489b4d8a8dc415af0) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4}
I20240501 20:11:07.903463 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(24cc6ff1a0104ac690c8d8f82c17200c) complete. Timing: real 0.231s user 0.064s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2347,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1620,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4023,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":13590,"threads_started":1,"wal-append.queue_time_us":6116}
I20240501 20:11:07.907135 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(24cc6ff1a0104ac690c8d8f82c17200c): 4581 bytes on disk
I20240501 20:11:07.907874 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(24cc6ff1a0104ac690c8d8f82c17200c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 20:11:07.968806 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(420b46edf5fc438fb01e822440bce8f1) complete. Timing: real 0.681s user 0.093s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":122579,"dirs.queue_time_us":51248,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":2017,"drs_written":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1007,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":226649,"threads_started":3,"wal-append.queue_time_us":28771}
I20240501 20:11:07.984889 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(420b46edf5fc438fb01e822440bce8f1): 4921 bytes on disk
I20240501 20:11:07.985980 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(420b46edf5fc438fb01e822440bce8f1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 20:11:07.989910 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(24cc6ff1a0104ac690c8d8f82c17200c): perf score=0.033585
I20240501 20:11:08.019687 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(f392e77df7ff4da489b4d8a8dc415af0) complete. Timing: real 0.796s user 0.088s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":19744,"dirs.queue_time_us":9922,"dirs.run_cpu_time_us":341,"dirs.run_wall_time_us":3079,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":978,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":93659,"threads_started":3,"wal-append.queue_time_us":28350}
I20240501 20:11:08.024814 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(24cc6ff1a0104ac690c8d8f82c17200c): perf score=0.033592
I20240501 20:11:08.126194 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(24cc6ff1a0104ac690c8d8f82c17200c) complete. Timing: real 0.097s user 0.039s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":1681,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":920,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":1408,"thread_start_us":6439,"threads_started":1,"wal-append.queue_time_us":6330}
I20240501 20:11:08.129169 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(f392e77df7ff4da489b4d8a8dc415af0): 4834 bytes on disk
I20240501 20:11:08.131186 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(f392e77df7ff4da489b4d8a8dc415af0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 20:11:08.139272 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(24cc6ff1a0104ac690c8d8f82c17200c): 4581 bytes on disk
I20240501 20:11:08.139936 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(24cc6ff1a0104ac690c8d8f82c17200c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 20:11:08.245214 24109 consensus_queue.cc:237] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:08.250159 21227 raft_consensus.cc:1261] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:08.250910 3380 consensus_queue.cc:1035] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:08.255638 23843 raft_consensus.cc:1261] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:08.256338 3380 consensus_queue.cc:1035] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:08.277133 3380 raft_consensus.cc:2932] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:08.285873 21228 raft_consensus.cc:2932] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:08.294799 20823 catalog_manager.cc:5484] T 4341f17784bc45359a9648ab47c39628 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:08.295107 23843 raft_consensus.cc:2932] T 4341f17784bc45359a9648ab47c39628 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
W20240501 20:11:08.351086 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 53858.3us
I20240501 20:11:08.385066 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 4341f17784bc45359a9648ab47c39628 with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51874
I20240501 20:11:08.395303 3767 tablet_replica.cc:330] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:11:08.395579 3767 raft_consensus.cc:2229] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:08.395799 3767 raft_consensus.cc:2258] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:08.396479 3767 ts_tablet_manager.cc:1875] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:08.417387 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(24cc6ff1a0104ac690c8d8f82c17200c) complete. Timing: real 0.427s user 0.074s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7018,"dirs.run_cpu_time_us":289,"dirs.run_wall_time_us":7931,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5892,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25861,"threads_started":2,"wal-append.queue_time_us":16374}
I20240501 20:11:08.420393 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(24cc6ff1a0104ac690c8d8f82c17200c): 4581 bytes on disk
I20240501 20:11:08.428092 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(24cc6ff1a0104ac690c8d8f82c17200c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 20:11:08.499984 3767 ts_tablet_manager.cc:1888] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:08.513213 3767 log.cc:1196] T 4341f17784bc45359a9648ab47c39628 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/4341f17784bc45359a9648ab47c39628
I20240501 20:11:08.520380 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 4341f17784bc45359a9648ab47c39628 (table rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]) successfully deleted
I20240501 20:11:09.231956 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:09.273579 21338 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 20:11:09.309018 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:09.333173 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:09.414397 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:09.438773 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:09.440675 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:09.484227 21077 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:09.826503 21096 consensus_queue.cc:237] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:09.830756 21227 raft_consensus.cc:1261] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:09.831343 3791 consensus_queue.cc:1035] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:09.832996 24108 raft_consensus.cc:1261] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:09.833693 3791 consensus_queue.cc:1035] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:09.836964 3791 raft_consensus.cc:2932] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:09.842494 21227 raft_consensus.cc:2932] T e1476e8f4c9343928f68365fd2772f0f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:09.854691 20819 catalog_manager.cc:5484] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:09.865648 24108 raft_consensus.cc:2932] T e1476e8f4c9343928f68365fd2772f0f P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:09.901064 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet e1476e8f4c9343928f68365fd2772f0f with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42940
I20240501 20:11:09.914186 3874 tablet_replica.cc:330] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:11:09.914618 3874 raft_consensus.cc:2229] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:09.915081 3874 raft_consensus.cc:2258] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:09.916221 3874 ts_tablet_manager.cc:1875] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:09.923112 3874 ts_tablet_manager.cc:1888] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:11:09.923383 3874 log.cc:1196] T e1476e8f4c9343928f68365fd2772f0f P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/e1476e8f4c9343928f68365fd2772f0f
I20240501 20:11:09.928179 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet e1476e8f4c9343928f68365fd2772f0f (table rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]) successfully deleted
I20240501 20:11:10.228868 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:10.250707 21338 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 20:11:10.251327 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:10.340638 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:10.343802 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:10.376166 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:10.415431 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(021fed14d5a140f3b42d359bae264f69): perf score=0.033369
I20240501 20:11:10.417479 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:10.425599 21077 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:10.436515 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(021fed14d5a140f3b42d359bae264f69): perf score=0.033385
I20240501 20:11:10.440896 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(aebb2a23076449fd9db492ed3e3b0cbf): perf score=0.033375
I20240501 20:11:10.456703 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(aebb2a23076449fd9db492ed3e3b0cbf): perf score=0.033381
W20240501 20:11:10.482162 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 63191.5us
I20240501 20:11:10.602490 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(021fed14d5a140f3b42d359bae264f69) complete. Timing: real 0.182s user 0.071s sys 0.044s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":65565,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":4219,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":877,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":73107,"threads_started":2,"wal-append.queue_time_us":7523}
I20240501 20:11:10.605485 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(021fed14d5a140f3b42d359bae264f69): 4921 bytes on disk
I20240501 20:11:10.608050 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(021fed14d5a140f3b42d359bae264f69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 20:11:10.661350 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(021fed14d5a140f3b42d359bae264f69) complete. Timing: real 0.224s user 0.050s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6438,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1553,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1086,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":41180,"threads_started":2,"wal-append.queue_time_us":17641}
I20240501 20:11:10.664311 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(021fed14d5a140f3b42d359bae264f69): 4921 bytes on disk
I20240501 20:11:10.667587 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(021fed14d5a140f3b42d359bae264f69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":144,"lbm_reads_lt_1ms":4}
I20240501 20:11:10.670542 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(c1e295a46a174f3f8673e03f57890b58): perf score=0.033437
I20240501 20:11:10.672628 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(aebb2a23076449fd9db492ed3e3b0cbf) complete. Timing: real 0.216s user 0.047s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6844,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":13258,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3663,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":14197,"threads_started":2,"wal-append.queue_time_us":7115}
I20240501 20:11:10.673939 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(aebb2a23076449fd9db492ed3e3b0cbf) complete. Timing: real 0.233s user 0.038s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18597,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":7449,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1003,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":42803,"threads_started":2,"wal-append.queue_time_us":11410}
I20240501 20:11:10.675387 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(aebb2a23076449fd9db492ed3e3b0cbf): 4581 bytes on disk
I20240501 20:11:10.676167 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(aebb2a23076449fd9db492ed3e3b0cbf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 20:11:10.676301 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(aebb2a23076449fd9db492ed3e3b0cbf): 4581 bytes on disk
I20240501 20:11:10.676831 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(aebb2a23076449fd9db492ed3e3b0cbf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 20:11:10.679378 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(c1e295a46a174f3f8673e03f57890b58): perf score=0.033439
I20240501 20:11:10.762377 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(c1e295a46a174f3f8673e03f57890b58) complete. Timing: real 0.092s user 0.033s sys 0.031s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1384,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":977,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":20177,"threads_started":1,"wal-append.queue_time_us":19982}
I20240501 20:11:10.765439 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(c1e295a46a174f3f8673e03f57890b58): 4834 bytes on disk
I20240501 20:11:10.766192 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(c1e295a46a174f3f8673e03f57890b58) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240501 20:11:10.766460 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(c1e295a46a174f3f8673e03f57890b58) complete. Timing: real 0.085s user 0.044s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1465,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6432,"threads_started":1,"wal-append.queue_time_us":6326}
I20240501 20:11:10.769217 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(c1e295a46a174f3f8673e03f57890b58): 4834 bytes on disk
I20240501 20:11:10.769871 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(c1e295a46a174f3f8673e03f57890b58) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 20:11:10.803508 23710 consensus_queue.cc:237] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:10.806228 20963 raft_consensus.cc:1261] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:10.806548 24109 raft_consensus.cc:1261] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:10.806764 3286 consensus_queue.cc:1035] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:10.807256 3755 consensus_queue.cc:1035] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:10.821882 3286 raft_consensus.cc:2932] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:10.822357 24109 raft_consensus.cc:2932] T 024bb310955b4a2e80d908b8c0a8e4c1 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:10.822628 20965 raft_consensus.cc:2932] T 024bb310955b4a2e80d908b8c0a8e4c1 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:10.828940 20822 catalog_manager.cc:5484] T 024bb310955b4a2e80d908b8c0a8e4c1 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:10.843178 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 024bb310955b4a2e80d908b8c0a8e4c1 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:11:10.856051 3965 tablet_replica.cc:330] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:11:10.856452 3965 raft_consensus.cc:2229] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:10.856787 3965 raft_consensus.cc:2258] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:10.857755 3965 ts_tablet_manager.cc:1875] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:10.865399 3965 ts_tablet_manager.cc:1888] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:10.865574 3965 log.cc:1196] T 024bb310955b4a2e80d908b8c0a8e4c1 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/024bb310955b4a2e80d908b8c0a8e4c1
I20240501 20:11:10.866325 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 024bb310955b4a2e80d908b8c0a8e4c1 (table rebalancer_extra_table_32 [id=634e55aaede844dfb0f7435e0bbb2cb9]) successfully deleted
I20240501 20:11:11.247804 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:11.268731 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:11.279178 21338 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 20:11:11.380887 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:11.397852 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:11.413172 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:11.483786 21077 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:11.508764 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:11.703044 21357 consensus_queue.cc:237] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:11.705284 23710 raft_consensus.cc:1261] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.705410 20965 raft_consensus.cc:1261] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.705885 3283 consensus_queue.cc:1035] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.706221 3928 consensus_queue.cc:1035] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.710014 3928 raft_consensus.cc:2932] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:11.710515 23710 raft_consensus.cc:2932] T ca03186ff8b342dd958b205125b14177 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:11.718611 20821 catalog_manager.cc:5484] T ca03186ff8b342dd958b205125b14177 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:11.718705 20965 raft_consensus.cc:2932] T ca03186ff8b342dd958b205125b14177 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:11.739562 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet ca03186ff8b342dd958b205125b14177 with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51330
I20240501 20:11:11.740272 4058 tablet_replica.cc:330] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:11:11.740659 4058 raft_consensus.cc:2229] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:11.740932 4058 raft_consensus.cc:2258] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:11.741647 4058 ts_tablet_manager.cc:1875] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:11.747404 4058 ts_tablet_manager.cc:1888] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:11:11.747521 4058 log.cc:1196] T ca03186ff8b342dd958b205125b14177 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/ca03186ff8b342dd958b205125b14177
I20240501 20:11:11.748226 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet ca03186ff8b342dd958b205125b14177 (table rebalance_test_table_17 [id=6d6f50d8217e43c18f819ef459761c81]) successfully deleted
I20240501 20:11:12.031723 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:12.057400 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:12.068128 21338 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 20:11:12.128381 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:12.152535 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:12.185587 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:12.192772 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:12.230450 21077 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:12.339967 21357 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b70e144824074927bb0f4eafcdeadcec"
dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55728
I20240501 20:11:12.340201 21357 raft_consensus.cc:595] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.349359 4049 raft_consensus.cc:979] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: : Instructing follower feac73db0a3b4aafa374ca5dfd322b08 to start an election
I20240501 20:11:12.349576 4049 raft_consensus.cc:1067] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Signalling peer feac73db0a3b4aafa374ca5dfd322b08 to start an election
I20240501 20:11:12.350318 21096 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b70e144824074927bb0f4eafcdeadcec"
dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
from {username='slave'} at 127.13.251.132:47489
I20240501 20:11:12.350512 21096 raft_consensus.cc:482] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:12.350607 21096 raft_consensus.cc:3037] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.352846 21096 raft_consensus.cc:504] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:12.353926 21357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70e144824074927bb0f4eafcdeadcec" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:11:12.354204 21357 raft_consensus.cc:3032] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:12.354290 21357 raft_consensus.cc:729] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:11:12.354338 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70e144824074927bb0f4eafcdeadcec" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:11:12.354521 24108 raft_consensus.cc:3037] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.354645 21357 consensus_queue.cc:260] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:12.355190 21357 raft_consensus.cc:3037] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.356690 24108 raft_consensus.cc:2454] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:11:12.357051 21096 leader_election.cc:288] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Requested vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:11:12.359207 21357 raft_consensus.cc:2454] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:11:12.359784 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70e144824074927bb0f4eafcdeadcec" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4af096f5a6a9440a94f6682b399efe20"
I20240501 20:11:12.360056 21227 raft_consensus.cc:3037] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.362496 21227 raft_consensus.cc:2454] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:11:12.363091 21034 leader_election.cc:302] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:11:12.363440 3791 raft_consensus.cc:2781] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:12.363602 3791 raft_consensus.cc:686] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 2 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:11:12.363888 3791 consensus_queue.cc:237] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:12.369093 20824 catalog_manager.cc:5484] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 1 to 2, leader changed from 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 2 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
W20240501 20:11:12.567205 23782 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 65530.7us
I20240501 20:11:12.738075 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:12.757458 24108 raft_consensus.cc:1261] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:11:12.758358 3791 consensus_queue.cc:1035] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.784756 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:12.805653 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:12.844388 21358 raft_consensus.cc:1261] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:11:12.845160 3791 consensus_queue.cc:1035] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:11:12.856526 4187 log.cc:924] Time spent T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240501 20:11:12.858008 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:12.882210 21227 raft_consensus.cc:1261] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:11:12.887979 3630 consensus_queue.cc:1035] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.889747 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:12.916779 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:12.956804 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:12.991008 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:13.374182 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:13.394201 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:13.395988 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(d25f96315839453d8cd5458417404e34): perf score=0.033347
I20240501 20:11:13.396852 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:13.429834 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:13.456512 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:13.460201 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(e5da8abdb3014753ac3cbef1d622069e): perf score=0.033362
I20240501 20:11:13.461957 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:13.501937 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:13.552829 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(d25f96315839453d8cd5458417404e34): perf score=0.033388
I20240501 20:11:13.555032 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(d25f96315839453d8cd5458417404e34): perf score=0.033387
I20240501 20:11:13.586200 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:13.610586 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(d25f96315839453d8cd5458417404e34) complete. Timing: real 0.195s user 0.046s sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54374,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":7785,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2119,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71686,"threads_started":2,"wal-append.queue_time_us":8938}
I20240501 20:11:13.630174 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(e5da8abdb3014753ac3cbef1d622069e): perf score=0.033404
I20240501 20:11:13.671474 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(e5da8abdb3014753ac3cbef1d622069e) complete. Timing: real 0.211s user 0.037s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":19707,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":3638,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":787,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":39472,"threads_started":2,"wal-append.queue_time_us":11014}
I20240501 20:11:13.674338 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(e5da8abdb3014753ac3cbef1d622069e): 4834 bytes on disk
I20240501 20:11:13.674957 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(e5da8abdb3014753ac3cbef1d622069e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 20:11:13.744650 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(e5da8abdb3014753ac3cbef1d622069e) complete. Timing: real 0.114s user 0.057s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1544,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1036,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":14657,"threads_started":1,"wal-append.queue_time_us":14758}
I20240501 20:11:13.746845 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(d25f96315839453d8cd5458417404e34): 4921 bytes on disk
I20240501 20:11:13.748478 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(d25f96315839453d8cd5458417404e34) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 20:11:13.751933 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(e5da8abdb3014753ac3cbef1d622069e): 4834 bytes on disk
I20240501 20:11:13.754339 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(e5da8abdb3014753ac3cbef1d622069e) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 20:11:13.812553 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(d25f96315839453d8cd5458417404e34) complete. Timing: real 0.257s user 0.054s sys 0.047s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":63733,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":31552,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":908,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":65700,"threads_started":2,"wal-append.queue_time_us":2161}
I20240501 20:11:13.814656 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(d25f96315839453d8cd5458417404e34): 4921 bytes on disk
I20240501 20:11:13.815171 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(d25f96315839453d8cd5458417404e34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 20:11:13.817111 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(2b4384496d9048259bf353bd58a15008): perf score=0.033458
I20240501 20:11:13.832101 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(d25f96315839453d8cd5458417404e34) complete. Timing: real 0.272s user 0.051s sys 0.033s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":34163,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":7966,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":63740,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":41982,"threads_started":2,"wal-append.queue_time_us":8044}
I20240501 20:11:13.836192 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(d25f96315839453d8cd5458417404e34): 4921 bytes on disk
I20240501 20:11:13.836979 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(d25f96315839453d8cd5458417404e34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240501 20:11:13.840725 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(2b4384496d9048259bf353bd58a15008): perf score=0.033462
I20240501 20:11:13.900069 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(2b4384496d9048259bf353bd58a15008) complete. Timing: real 0.080s user 0.067s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1363,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1066,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":303,"threads_started":1,"wal-append.queue_time_us":448}
I20240501 20:11:13.905050 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(2b4384496d9048259bf353bd58a15008): 4581 bytes on disk
I20240501 20:11:13.905695 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(2b4384496d9048259bf353bd58a15008) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 20:11:13.916931 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(2b4384496d9048259bf353bd58a15008) complete. Timing: real 0.076s user 0.045s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1366,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":894,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3871,"threads_started":1,"wal-append.queue_time_us":3984}
I20240501 20:11:13.918959 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(2b4384496d9048259bf353bd58a15008): 4581 bytes on disk
I20240501 20:11:13.919622 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(2b4384496d9048259bf353bd58a15008) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 20:11:13.928076 21227 consensus_queue.cc:237] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:13.930449 23709 raft_consensus.cc:1261] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:13.930430 21357 raft_consensus.cc:1261] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 4af096f5a6a9440a94f6682b399efe20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:13.931178 4271 consensus_queue.cc:1035] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:13.931480 4270 consensus_queue.cc:1035] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:13.949656 4271 raft_consensus.cc:2932] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:13.954653 23709 raft_consensus.cc:2932] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:13.959165 20824 catalog_manager.cc:5484] T 24cc6ff1a0104ac690c8d8f82c17200c P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "4af096f5a6a9440a94f6682b399efe20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:13.966259 21357 raft_consensus.cc:2932] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:13.990650 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet 24cc6ff1a0104ac690c8d8f82c17200c with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51330
I20240501 20:11:14.004475 4304 tablet_replica.cc:330] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:11:14.004711 4304 raft_consensus.cc:2229] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:14.005021 4304 raft_consensus.cc:2258] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:14.006119 4304 ts_tablet_manager.cc:1875] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:14.028035 4304 ts_tablet_manager.cc:1888] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:11:14.028192 4304 log.cc:1196] T 24cc6ff1a0104ac690c8d8f82c17200c P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/24cc6ff1a0104ac690c8d8f82c17200c
I20240501 20:11:14.029114 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet 24cc6ff1a0104ac690c8d8f82c17200c (table rebalance_test_table_1 [id=16c4863630ae4b2dae19f7c9d21c7fae]) successfully deleted
I20240501 20:11:14.335408 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:14.362908 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:14.376593 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:14.401801 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:14.442016 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:14.454895 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:14.468333 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:14.525856 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:14.650959 23842 consensus_queue.cc:237] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:14.653561 23710 raft_consensus.cc:1261] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:14.653767 21228 raft_consensus.cc:1261] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:14.654273 4165 consensus_queue.cc:1035] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:14.654561 4161 consensus_queue.cc:1035] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:14.656900 4165 raft_consensus.cc:2932] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:14.657382 21228 raft_consensus.cc:2932] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:14.663282 20822 catalog_manager.cc:5484] T cf1ac1f2f3d642e9a9c785bb4963af1f P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:14.668128 23709 raft_consensus.cc:2932] T cf1ac1f2f3d642e9a9c785bb4963af1f P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:14.697376 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1ac1f2f3d642e9a9c785bb4963af1f with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51874
I20240501 20:11:14.708992 4390 tablet_replica.cc:330] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:11:14.709444 4390 raft_consensus.cc:2229] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:14.709769 4390 raft_consensus.cc:2258] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:14.710686 4390 ts_tablet_manager.cc:1875] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:14.716643 4390 ts_tablet_manager.cc:1888] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:14.716820 4390 log.cc:1196] T cf1ac1f2f3d642e9a9c785bb4963af1f P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/cf1ac1f2f3d642e9a9c785bb4963af1f
I20240501 20:11:14.717628 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet cf1ac1f2f3d642e9a9c785bb4963af1f (table rebalancer_extra_table_16 [id=51de3ca9566d47339bba5fe606031ebe]) successfully deleted
I20240501 20:11:14.974579 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:14.984261 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:14.989213 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:15.058475 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:15.071782 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:15.090003 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:15.121774 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:15.128940 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:15.288146 23842 consensus_queue.cc:237] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:15.289939 24108 raft_consensus.cc:1261] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:15.289939 21227 raft_consensus.cc:1261] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:15.290441 4161 consensus_queue.cc:1035] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:15.290722 4269 consensus_queue.cc:1035] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:15.293677 4161 raft_consensus.cc:2932] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:15.294279 24108 raft_consensus.cc:2932] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:15.300748 20820 catalog_manager.cc:5484] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New cstate: current_term: 1 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:15.317999 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8 with delete_type TABLET_DATA_TOMBSTONED (TS 744c5a532e724391830c0785d32eb2fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51874
I20240501 20:11:15.318626 4477 tablet_replica.cc:330] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb: stopping tablet replica
I20240501 20:11:15.319782 21227 raft_consensus.cc:2932] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 744c5a532e724391830c0785d32eb2fb (127.13.251.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:15.321880 4477 raft_consensus.cc:2229] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:15.322204 4477 raft_consensus.cc:2258] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:15.324973 4477 ts_tablet_manager.cc:1875] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:15.334725 4477 ts_tablet_manager.cc:1888] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:15.334838 4477 log.cc:1196] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P 744c5a532e724391830c0785d32eb2fb: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-6/wal/wals/224f83ad3d514a7fa5cdfa4e70ff2fd8
I20240501 20:11:15.335525 20810 catalog_manager.cc:4830] TS 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019): tablet 224f83ad3d514a7fa5cdfa4e70ff2fd8 (table rebalancer_extra_table_30 [id=9697904f8b1244aa99ea4c0afe869847]) successfully deleted
I20240501 20:11:15.640296 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:15.655512 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:15.659747 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:15.681171 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:15.687047 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(10a4d24c5c844c8c9ccd191b1a2b0b00): perf score=0.033369
I20240501 20:11:15.695700 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(554b1d57dbb541508afaf8dbb77cbdad): perf score=0.033366
I20240501 20:11:15.702098 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:15.713153 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(10a4d24c5c844c8c9ccd191b1a2b0b00): perf score=0.033373
I20240501 20:11:15.762995 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:15.774940 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:15.778214 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(554b1d57dbb541508afaf8dbb77cbdad): perf score=0.033393
I20240501 20:11:15.820479 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(10a4d24c5c844c8c9ccd191b1a2b0b00) complete. Timing: real 0.107s user 0.037s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6425,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":6094,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":857,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":21122,"threads_started":2,"wal-append.queue_time_us":13184}
I20240501 20:11:15.823657 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(10a4d24c5c844c8c9ccd191b1a2b0b00): 4581 bytes on disk
I20240501 20:11:15.826874 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(10a4d24c5c844c8c9ccd191b1a2b0b00) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":2686,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240501 20:11:15.844936 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:15.904605 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(554b1d57dbb541508afaf8dbb77cbdad) complete. Timing: real 0.126s user 0.046s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":22635,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":2274,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":784,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":39001,"threads_started":2,"wal-append.queue_time_us":14098}
I20240501 20:11:15.908551 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(72726d6b786e4c72ace7359101cec949): perf score=0.033426
I20240501 20:11:15.925694 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(554b1d57dbb541508afaf8dbb77cbdad) complete. Timing: real 0.230s user 0.039s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":20220,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":2585,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":738,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":22459,"threads_started":2,"wal-append.queue_time_us":1535}
I20240501 20:11:15.929123 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(72726d6b786e4c72ace7359101cec949): perf score=0.033426
I20240501 20:11:15.932010 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(10a4d24c5c844c8c9ccd191b1a2b0b00) complete. Timing: real 0.245s user 0.041s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":19440,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":2661,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1005,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25751,"threads_started":2,"wal-append.queue_time_us":6321}
I20240501 20:11:15.934948 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(10a4d24c5c844c8c9ccd191b1a2b0b00): 4581 bytes on disk
I20240501 20:11:15.935765 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(10a4d24c5c844c8c9ccd191b1a2b0b00) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":292,"lbm_reads_lt_1ms":4}
I20240501 20:11:16.005625 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(72726d6b786e4c72ace7359101cec949) complete. Timing: real 0.097s user 0.051s sys 0.034s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1440,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":817,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13710,"threads_started":1,"wal-append.queue_time_us":13839}
I20240501 20:11:16.009692 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(554b1d57dbb541508afaf8dbb77cbdad): 4921 bytes on disk
I20240501 20:11:16.010345 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(554b1d57dbb541508afaf8dbb77cbdad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240501 20:11:16.015863 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(72726d6b786e4c72ace7359101cec949): 4834 bytes on disk
I20240501 20:11:16.017140 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(72726d6b786e4c72ace7359101cec949) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":629,"lbm_reads_lt_1ms":4}
I20240501 20:11:16.069977 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(72726d6b786e4c72ace7359101cec949) complete. Timing: real 0.134s user 0.053s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1588,"drs_written":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4,"lbm_write_time_us":983,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":287,"threads_started":1,"wal-append.queue_time_us":415}
I20240501 20:11:16.072222 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(554b1d57dbb541508afaf8dbb77cbdad): 4921 bytes on disk
I20240501 20:11:16.074853 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(554b1d57dbb541508afaf8dbb77cbdad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 20:11:16.077227 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(72726d6b786e4c72ace7359101cec949): 4834 bytes on disk
I20240501 20:11:16.077832 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(72726d6b786e4c72ace7359101cec949) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
W20240501 20:11:16.080209 21031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 44377.8us
I20240501 20:11:16.145440 24108 consensus_queue.cc:237] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:16.147373 20965 raft_consensus.cc:1261] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:16.147380 23710 raft_consensus.cc:1261] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:16.147917 1855 consensus_queue.cc:1035] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:16.148202 4180 consensus_queue.cc:1035] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:16.151119 4180 raft_consensus.cc:2932] T 2d2e66020ffb438fad69c2ea73762aa7 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:16.155905 23710 raft_consensus.cc:2932] T 2d2e66020ffb438fad69c2ea73762aa7 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:16.159487 20965 raft_consensus.cc:2932] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:16.165455 20821 catalog_manager.cc:5484] T 2d2e66020ffb438fad69c2ea73762aa7 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:16.181951 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2e66020ffb438fad69c2ea73762aa7 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:11:16.195416 4589 tablet_replica.cc:330] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:11:16.195776 4589 raft_consensus.cc:2229] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:16.196015 4589 raft_consensus.cc:2258] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:16.196774 4589 ts_tablet_manager.cc:1875] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:16.203064 4589 ts_tablet_manager.cc:1888] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:16.203202 4589 log.cc:1196] T 2d2e66020ffb438fad69c2ea73762aa7 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/2d2e66020ffb438fad69c2ea73762aa7
I20240501 20:11:16.204185 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 2d2e66020ffb438fad69c2ea73762aa7 (table rebalancer_extra_table_8 [id=6f73adbc46754907b69a62aa47ba2d82]) successfully deleted
I20240501 20:11:16.512107 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:16.526067 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:16.532444 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:16.572059 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:16.575635 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:16.601889 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:16.636953 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:16.672798 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:16.811203 21098 consensus_queue.cc:237] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:16.813911 20963 raft_consensus.cc:1261] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:16.814076 23842 raft_consensus.cc:1261] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:16.814543 3630 consensus_queue.cc:1035] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.814846 4561 consensus_queue.cc:1035] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.817566 4561 raft_consensus.cc:2932] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:16.818259 23842 raft_consensus.cc:2932] T 998f71aeae984fd5b866cb866bd4e87a P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:16.818325 20963 raft_consensus.cc:2932] T 998f71aeae984fd5b866cb866bd4e87a P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:16.825956 20821 catalog_manager.cc:5484] T 998f71aeae984fd5b866cb866bd4e87a P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 2 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:16.847913 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet 998f71aeae984fd5b866cb866bd4e87a with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42940
I20240501 20:11:16.848634 4687 tablet_replica.cc:330] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:11:16.848949 4687 raft_consensus.cc:2229] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:16.849198 4687 raft_consensus.cc:2258] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:16.850991 4687 ts_tablet_manager.cc:1875] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:16.858405 4687 ts_tablet_manager.cc:1888] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:11:16.858605 4687 log.cc:1196] T 998f71aeae984fd5b866cb866bd4e87a P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/998f71aeae984fd5b866cb866bd4e87a
I20240501 20:11:16.860023 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet 998f71aeae984fd5b866cb866bd4e87a (table rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]) successfully deleted
I20240501 20:11:16.875909 21098 consensus_queue.cc:237] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:16.878324 21228 raft_consensus.cc:1261] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:16.878322 21357 raft_consensus.cc:1261] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:16.878775 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): Couldn't send request to peer c6dcaa8efff642dcb7c2c6e8be3a65f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d25f96315839453d8cd5458417404e34. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:16.879070 4464 consensus_queue.cc:1035] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:16.879384 3630 consensus_queue.cc:1035] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:16.882715 3630 raft_consensus.cc:2932] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:16.883383 21357 raft_consensus.cc:2932] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:16.887296 21228 raft_consensus.cc:2932] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:16.894480 20819 catalog_manager.cc:5484] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index -1 to 4, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:16.928084 20963 consensus_queue.cc:237] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
W20240501 20:11:16.932571 20902 consensus_peers.cc:485] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa02aed57a7a4e6a95b149d2078dc265. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:16.932669 21098 raft_consensus.cc:1261] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:16.933514 4189 consensus_queue.cc:1035] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:16.935322 21357 raft_consensus.cc:1261] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:16.935963 4189 consensus_queue.cc:1035] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:16.944666 4189 raft_consensus.cc:2932] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:16.950137 21098 raft_consensus.cc:2932] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:16.966583 20825 catalog_manager.cc:5484] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:16.975910 21357 raft_consensus.cc:2932] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:17.024156 21357 consensus_queue.cc:237] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:17.026643 21098 raft_consensus.cc:1261] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:17.026937 21228 raft_consensus.cc:1261] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:17.027225 4453 consensus_queue.cc:1035] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:17.027454 21293 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155): Couldn't send request to peer 51bf640e895b4e5eb91bdf7509ea7e46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed9b47caf0a041fd962e4f18b0de5ff2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.027490 4162 consensus_queue.cc:1035] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:17.051980 4162 raft_consensus.cc:2932] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:17.059142 21098 raft_consensus.cc:2932] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:17.065053 21228 raft_consensus.cc:2932] T ed9b47caf0a041fd962e4f18b0de5ff2 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:17.066571 20821 catalog_manager.cc:5484] T ed9b47caf0a041fd962e4f18b0de5ff2 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 4, VOTER 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133) added. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:17.297436 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:17.308014 4793 ts_tablet_manager.cc:902] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:11:17.325724 4793 tablet_copy_client.cc:308] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:11:17.327023 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:17.333602 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet d25f96315839453d8cd5458417404e34 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 ({username='slave'} at 127.13.251.134:55907)
I20240501 20:11:17.333768 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet d25f96315839453d8cd5458417404e34 from peer c6dcaa8efff642dcb7c2c6e8be3a65f7 at {username='slave'} at 127.13.251.134:55907: session id = c6dcaa8efff642dcb7c2c6e8be3a65f7-d25f96315839453d8cd5458417404e34
I20240501 20:11:17.337271 21118 tablet_copy_source_session.cc:205] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:11:17.342015 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f96315839453d8cd5458417404e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:17.351223 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:17.351258 4793 tablet_copy_client.cc:783] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 5 data blocks...
I20240501 20:11:17.360109 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:17.367619 4793 tablet_copy_client.cc:647] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:17.370532 4793 tablet_copy_client.cc:523] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:17.374679 4793 tablet_bootstrap.cc:492] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap starting.
I20240501 20:11:17.379782 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:17.400537 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:17.421248 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:17.456928 4793 tablet_bootstrap.cc:492] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:17.457407 4793 tablet_bootstrap.cc:492] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Bootstrap complete.
I20240501 20:11:17.461733 4793 ts_tablet_manager.cc:1367] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent bootstrapping tablet: real 0.087s user 0.034s sys 0.002s
I20240501 20:11:17.462644 4793 raft_consensus.cc:348] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:17.462898 4793 raft_consensus.cc:729] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Initialized, Role: FOLLOWER
I20240501 20:11:17.463297 4793 consensus_queue.cc:260] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:17.465902 4793 ts_tablet_manager.cc:1398] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:11:17.466485 21118 tablet_copy_service.cc:342] P feac73db0a3b4aafa374ca5dfd322b08: Request end of tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-d25f96315839453d8cd5458417404e34 received from {username='slave'} at 127.13.251.134:55907
I20240501 20:11:17.466603 21118 tablet_copy_service.cc:433] P feac73db0a3b4aafa374ca5dfd322b08: ending tablet copy session c6dcaa8efff642dcb7c2c6e8be3a65f7-d25f96315839453d8cd5458417404e34 on tablet d25f96315839453d8cd5458417404e34 with peer c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:11:17.484246 4805 ts_tablet_manager.cc:902] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:11:17.514715 4805 tablet_copy_client.cc:308] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.129:41259
I20240501 20:11:17.515419 20987 tablet_copy_service.cc:140] P cda19d6a81c0473db302535ce2ada7f0: Received BeginTabletCopySession request for tablet aa02aed57a7a4e6a95b149d2078dc265 from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:53673)
I20240501 20:11:17.515578 20987 tablet_copy_service.cc:161] P cda19d6a81c0473db302535ce2ada7f0: Beginning new tablet copy session on tablet aa02aed57a7a4e6a95b149d2078dc265 from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:53673: session id = 51bf640e895b4e5eb91bdf7509ea7e46-aa02aed57a7a4e6a95b149d2078dc265
I20240501 20:11:17.518445 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:17.518394 20987 tablet_copy_source_session.cc:205] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:17.525960 4805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa02aed57a7a4e6a95b149d2078dc265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:17.532773 4805 tablet_copy_client.cc:783] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:17.533006 4805 tablet_copy_client.cc:647] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:17.535805 4805 tablet_copy_client.cc:523] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:17.538601 4805 tablet_bootstrap.cc:492] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:11:17.621944 4805 tablet_bootstrap.cc:492] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:17.622400 4805 tablet_bootstrap.cc:492] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:11:17.635138 4813 ts_tablet_manager.cc:902] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: Initiating tablet copy from peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:11:17.639901 4805 ts_tablet_manager.cc:1367] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.101s user 0.052s sys 0.017s
I20240501 20:11:17.640604 4805 raft_consensus.cc:348] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:17.640823 4805 raft_consensus.cc:729] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:11:17.641057 4805 consensus_queue.cc:260] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:17.643306 4805 ts_tablet_manager.cc:1398] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:17.643896 20987 tablet_copy_service.cc:342] P cda19d6a81c0473db302535ce2ada7f0: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-aa02aed57a7a4e6a95b149d2078dc265 received from {username='slave'} at 127.13.251.133:53673
I20240501 20:11:17.644016 20987 tablet_copy_service.cc:433] P cda19d6a81c0473db302535ce2ada7f0: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-aa02aed57a7a4e6a95b149d2078dc265 on tablet aa02aed57a7a4e6a95b149d2078dc265 with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:11:17.666631 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling UndoDeltaBlockGCOp(d25f96315839453d8cd5458417404e34): 4921 bytes on disk
I20240501 20:11:17.667984 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: UndoDeltaBlockGCOp(d25f96315839453d8cd5458417404e34) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 20:11:17.696130 4813 tablet_copy_client.cc:308] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.132:46399
I20240501 20:11:17.696748 21378 tablet_copy_service.cc:140] P 61380023001a4ee9ada9ec2c6c1b7159: Received BeginTabletCopySession request for tablet ed9b47caf0a041fd962e4f18b0de5ff2 from peer 51bf640e895b4e5eb91bdf7509ea7e46 ({username='slave'} at 127.13.251.133:37097)
I20240501 20:11:17.696875 21378 tablet_copy_service.cc:161] P 61380023001a4ee9ada9ec2c6c1b7159: Beginning new tablet copy session on tablet ed9b47caf0a041fd962e4f18b0de5ff2 from peer 51bf640e895b4e5eb91bdf7509ea7e46 at {username='slave'} at 127.13.251.133:37097: session id = 51bf640e895b4e5eb91bdf7509ea7e46-ed9b47caf0a041fd962e4f18b0de5ff2
I20240501 20:11:17.698926 21378 tablet_copy_source_session.cc:205] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:17.699999 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9b47caf0a041fd962e4f18b0de5ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:17.705420 4813 tablet_copy_client.cc:783] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:17.705619 4813 tablet_copy_client.cc:647] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:17.707221 4813 tablet_copy_client.cc:523] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:17.710572 4813 tablet_bootstrap.cc:492] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap starting.
I20240501 20:11:17.757848 4813 tablet_bootstrap.cc:492] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:17.758257 4813 tablet_bootstrap.cc:492] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: Bootstrap complete.
I20240501 20:11:17.758514 4813 ts_tablet_manager.cc:1367] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent bootstrapping tablet: real 0.048s user 0.046s sys 0.000s
I20240501 20:11:17.759145 4813 raft_consensus.cc:348] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:17.759357 4813 raft_consensus.cc:729] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Initialized, Role: FOLLOWER
I20240501 20:11:17.759581 4813 consensus_queue.cc:260] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:17.760056 4813 ts_tablet_manager.cc:1398] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 20:11:17.761380 21378 tablet_copy_service.cc:342] P 61380023001a4ee9ada9ec2c6c1b7159: Request end of tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-ed9b47caf0a041fd962e4f18b0de5ff2 received from {username='slave'} at 127.13.251.133:37097
I20240501 20:11:17.761519 21378 tablet_copy_service.cc:433] P 61380023001a4ee9ada9ec2c6c1b7159: ending tablet copy session 51bf640e895b4e5eb91bdf7509ea7e46-ed9b47caf0a041fd962e4f18b0de5ff2 on tablet ed9b47caf0a041fd962e4f18b0de5ff2 with peer 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:11:17.895829 23842 raft_consensus.cc:1203] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:17.907474 23709 raft_consensus.cc:1203] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:18.020584 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:18.044515 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:18.053479 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:18.107245 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:18.129647 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:18.135377 23709 raft_consensus.cc:1203] T ed9b47caf0a041fd962e4f18b0de5ff2 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:18.157353 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:18.173164 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:18.257449 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 151137us
I20240501 20:11:18.281201 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:18.396018 20965 consensus_queue.cc:237] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:18.397972 21358 raft_consensus.cc:1261] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:18.398129 23709 raft_consensus.cc:1261] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:18.398598 4696 consensus_queue.cc:1035] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:18.399245 4696 consensus_queue.cc:1035] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:18.411705 4718 raft_consensus.cc:2932] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:18.412281 23709 raft_consensus.cc:2932] T aa02aed57a7a4e6a95b149d2078dc265 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:18.413264 21357 raft_consensus.cc:2932] T aa02aed57a7a4e6a95b149d2078dc265 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:18.418017 20820 catalog_manager.cc:5484] T aa02aed57a7a4e6a95b149d2078dc265 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: config changed from index 4 to 5, VOTER feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130) evicted. New cstate: current_term: 1 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:18.428030 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet aa02aed57a7a4e6a95b149d2078dc265 with delete_type TABLET_DATA_TOMBSTONED (TS feac73db0a3b4aafa374ca5dfd322b08 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51330
I20240501 20:11:18.428773 4912 tablet_replica.cc:330] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08: stopping tablet replica
I20240501 20:11:18.431212 4912 raft_consensus.cc:2229] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:18.431457 4912 raft_consensus.cc:2258] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:18.432253 4912 ts_tablet_manager.cc:1875] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:18.438468 4912 ts_tablet_manager.cc:1888] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:11:18.439342 4912 log.cc:1196] T aa02aed57a7a4e6a95b149d2078dc265 P feac73db0a3b4aafa374ca5dfd322b08: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa02aed57a7a4e6a95b149d2078dc265
I20240501 20:11:18.440423 20810 catalog_manager.cc:4830] TS feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271): tablet aa02aed57a7a4e6a95b149d2078dc265 (table rebalance_test_table_9 [id=2b8b47ab2719465a95cf6b6874b2de29]) successfully deleted
I20240501 20:11:18.790696 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:18.798477 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:18.818385 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:18.828217 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:18.854311 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:18.860985 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(43e80df9af0e40c5acc4dea95d7e95a8): perf score=0.033341
I20240501 20:11:18.863471 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(aa02aed57a7a4e6a95b149d2078dc265): perf score=0.033341
I20240501 20:11:18.906147 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:18.915937 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:18.958606 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:18.978134 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(43e80df9af0e40c5acc4dea95d7e95a8): perf score=0.033378
W20240501 20:11:18.984673 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 63472.9us
I20240501 20:11:19.054266 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(a115fe66d11143ea8e24aa891e4e2f33): perf score=0.033391
I20240501 20:11:19.106092 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(43e80df9af0e40c5acc4dea95d7e95a8) complete. Timing: real 0.244s user 0.049s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5940,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":43250,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1366,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":18345,"threads_started":2,"wal-append.queue_time_us":582}
I20240501 20:11:19.108881 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(aa02aed57a7a4e6a95b149d2078dc265): perf score=0.033407
I20240501 20:11:19.113571 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(aa02aed57a7a4e6a95b149d2078dc265) complete. Timing: real 0.250s user 0.042s sys 0.059s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66233,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":2723,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":787,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":66082,"threads_started":1}
I20240501 20:11:19.116509 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(aa02aed57a7a4e6a95b149d2078dc265): 4834 bytes on disk
I20240501 20:11:19.117558 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(aa02aed57a7a4e6a95b149d2078dc265) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":459,"lbm_reads_lt_1ms":4}
I20240501 20:11:19.198539 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(a115fe66d11143ea8e24aa891e4e2f33) complete. Timing: real 0.144s user 0.044s sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3970,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":7506,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":796,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":4316,"threads_started":2,"wal-append.queue_time_us":305}
I20240501 20:11:19.201787 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(a115fe66d11143ea8e24aa891e4e2f33): 4921 bytes on disk
I20240501 20:11:19.202664 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(a115fe66d11143ea8e24aa891e4e2f33) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":170,"lbm_reads_lt_1ms":4}
I20240501 20:11:19.230541 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(43e80df9af0e40c5acc4dea95d7e95a8) complete. Timing: real 0.252s user 0.064s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":17971,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":6430,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8014,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":29379,"threads_started":2,"wal-append.queue_time_us":10911}
I20240501 20:11:19.233238 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(43e80df9af0e40c5acc4dea95d7e95a8): 4581 bytes on disk
I20240501 20:11:19.235146 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(43e80df9af0e40c5acc4dea95d7e95a8) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
I20240501 20:11:19.239060 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(aa02aed57a7a4e6a95b149d2078dc265) complete. Timing: real 0.124s user 0.032s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1348,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":17007,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676}
I20240501 20:11:19.241278 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(aa02aed57a7a4e6a95b149d2078dc265): 4834 bytes on disk
I20240501 20:11:19.243170 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(aa02aed57a7a4e6a95b149d2078dc265) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 20:11:19.243799 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(a115fe66d11143ea8e24aa891e4e2f33): perf score=0.033445
I20240501 20:11:19.246340 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(43e80df9af0e40c5acc4dea95d7e95a8): 4581 bytes on disk
I20240501 20:11:19.247273 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(43e80df9af0e40c5acc4dea95d7e95a8) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":436,"lbm_reads_lt_1ms":4}
I20240501 20:11:19.260977 21098 consensus_queue.cc:237] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:19.263315 23843 raft_consensus.cc:1261] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:19.263345 21227 raft_consensus.cc:1261] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:19.263839 3630 consensus_queue.cc:1035] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:19.264206 4807 consensus_queue.cc:1035] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:19.269218 3630 raft_consensus.cc:2932] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:19.269767 23843 raft_consensus.cc:2932] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:19.278328 20820 catalog_manager.cc:5484] T d25f96315839453d8cd5458417404e34 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:19.288336 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet d25f96315839453d8cd5458417404e34 with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42940
I20240501 20:11:19.289531 5013 tablet_replica.cc:330] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:11:19.289911 5013 raft_consensus.cc:2229] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:19.290377 5013 raft_consensus.cc:2258] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:19.291812 5013 ts_tablet_manager.cc:1875] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:19.308373 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(a115fe66d11143ea8e24aa891e4e2f33) complete. Timing: real 0.064s user 0.038s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":1693,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1037,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6396,"threads_started":1,"wal-append.queue_time_us":6294}
I20240501 20:11:19.309031 21228 raft_consensus.cc:2932] T d25f96315839453d8cd5458417404e34 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:19.311106 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(a115fe66d11143ea8e24aa891e4e2f33): 4921 bytes on disk
I20240501 20:11:19.314867 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(a115fe66d11143ea8e24aa891e4e2f33) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240501 20:11:19.324719 5013 ts_tablet_manager.cc:1888] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:11:19.324934 5013 log.cc:1196] T d25f96315839453d8cd5458417404e34 P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/d25f96315839453d8cd5458417404e34
I20240501 20:11:19.325737 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet d25f96315839453d8cd5458417404e34 (table rebalance_test_table_5 [id=8cfc77a5ed3d431d82a47792ff54bab4]) successfully deleted
I20240501 20:11:19.590947 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:19.650808 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:19.668881 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:19.688889 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:19.702116 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:19.760569 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:19.784363 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:19.830842 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:19.989391 21357 consensus_queue.cc:237] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:19.991677 23709 raft_consensus.cc:1261] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:19.991689 21228 raft_consensus.cc:1261] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:19.992216 4740 consensus_queue.cc:1035] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:19.992579 4814 consensus_queue.cc:1035] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:19.997298 4740 raft_consensus.cc:2932] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:20.000247 23709 raft_consensus.cc:2932] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:20.008404 20821 catalog_manager.cc:5484] T 236ba60733e1416395346179be3b3888 P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:20.020537 21228 raft_consensus.cc:2932] T 236ba60733e1416395346179be3b3888 P 4af096f5a6a9440a94f6682b399efe20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:20.032444 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 236ba60733e1416395346179be3b3888 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:11:20.037151 5109 tablet_replica.cc:330] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:11:20.039554 5109 raft_consensus.cc:2229] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:20.040243 5109 raft_consensus.cc:2258] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:20.040886 5109 ts_tablet_manager.cc:1875] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:20.047942 5109 ts_tablet_manager.cc:1888] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:20.048097 5109 log.cc:1196] T 236ba60733e1416395346179be3b3888 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/236ba60733e1416395346179be3b3888
I20240501 20:11:20.048942 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 236ba60733e1416395346179be3b3888 (table rebalancer_extra_table_2 [id=7b7f7edaf5b142cbbcf91bc16eacaa7f]) successfully deleted
I20240501 20:11:20.414091 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:11:20.436796 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:20.439030 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:20.473547 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:20.517333 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:20.541651 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:20.593027 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:20.597774 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:20.761325 21096 consensus_queue.cc:237] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:20.763569 24107 raft_consensus.cc:1261] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:20.763589 21227 raft_consensus.cc:1261] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:20.764050 4807 consensus_queue.cc:1035] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:20.765980 4807 consensus_queue.cc:1035] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:20.766958 5008 raft_consensus.cc:2932] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:20.768074 21227 raft_consensus.cc:2932] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:20.771500 24107 raft_consensus.cc:2932] T b70e144824074927bb0f4eafcdeadcec P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:20.774691 20826 catalog_manager.cc:5484] T b70e144824074927bb0f4eafcdeadcec P 4af096f5a6a9440a94f6682b399efe20 reported cstate change: config changed from index 4 to 6, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) evicted. New cstate: current_term: 2 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:20.780746 21338 tablet_service.cc:1514] Processing DeleteTablet for tablet b70e144824074927bb0f4eafcdeadcec with delete_type TABLET_DATA_TOMBSTONED (TS 61380023001a4ee9ada9ec2c6c1b7159 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42940
I20240501 20:11:20.787994 5217 tablet_replica.cc:330] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: stopping tablet replica
I20240501 20:11:20.788245 5217 raft_consensus.cc:2229] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:20.788445 5217 raft_consensus.cc:2258] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:20.789438 5217 ts_tablet_manager.cc:1875] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:20.806398 5217 ts_tablet_manager.cc:1888] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:11:20.806660 5217 log.cc:1196] T b70e144824074927bb0f4eafcdeadcec P 61380023001a4ee9ada9ec2c6c1b7159: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-3/wal/wals/b70e144824074927bb0f4eafcdeadcec
I20240501 20:11:20.807961 20811 catalog_manager.cc:4830] TS 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): tablet b70e144824074927bb0f4eafcdeadcec (table rebalance_test_table_13 [id=91a237bddf274ab59ba4459cf28caf4b]) successfully deleted
I20240501 20:11:20.882737 21096 consensus_queue.cc:237] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:20.885123 23709 raft_consensus.cc:1261] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:11:20.885447 21034 consensus_peers.cc:485] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1cbbb090cb34465b6dac9c35cdba0dc. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:20.885892 5213 consensus_queue.cc:1035] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:20.887254 23842 raft_consensus.cc:1261] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:20.888321 5213 consensus_queue.cc:1035] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:20.891381 5008 raft_consensus.cc:2932] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:20.892028 23842 raft_consensus.cc:2932] T a1cbbb090cb34465b6dac9c35cdba0dc P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:20.897902 23709 raft_consensus.cc:2932] T a1cbbb090cb34465b6dac9c35cdba0dc P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:20.900806 20817 catalog_manager.cc:5484] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:20.941740 24107 consensus_queue.cc:237] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:20.944672 23842 raft_consensus.cc:1261] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:20.945176 23710 raft_consensus.cc:1261] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:20.945919 4974 consensus_queue.cc:1035] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:20.946384 4974 consensus_queue.cc:1035] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:11:20.950091 24044 consensus_peers.cc:485] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 726a7cd006104059b4b10e6eab407577. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:20.967300 4974 raft_consensus.cc:2932] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:20.968170 23842 raft_consensus.cc:2932] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:20.976037 23710 raft_consensus.cc:2932] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:20.980518 20826 catalog_manager.cc:5484] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:21.079355 21096 consensus_queue.cc:237] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } }
I20240501 20:11:21.082099 23710 raft_consensus.cc:1261] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:21.083017 23842 raft_consensus.cc:1261] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:21.083276 4807 consensus_queue.cc:1035] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:11:21.084542 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c3bdb74d3d140e397ce0b5bdef5361d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:21.084947 5008 consensus_queue.cc:1035] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:21.089963 5008 raft_consensus.cc:2932] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:21.092042 23842 raft_consensus.cc:2932] T 6c3bdb74d3d140e397ce0b5bdef5361d P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:21.094990 23710 raft_consensus.cc:2932] T 6c3bdb74d3d140e397ce0b5bdef5361d P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:21.097489 20826 catalog_manager.cc:5484] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: config changed from index -1 to 2, VOTER 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) added. New cstate: current_term: 1 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } }
I20240501 20:11:21.232614 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(75fa04c8726c4f2e932740cc6fbdcdd0): perf score=0.033343
I20240501 20:11:21.271929 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(4587a9d5392742b58ec8d64619f2dda1): perf score=0.033354
I20240501 20:11:21.337545 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(75fa04c8726c4f2e932740cc6fbdcdd0) complete. Timing: real 0.105s user 0.054s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10496,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1953,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":20105,"threads_started":2,"wal-append.queue_time_us":9597}
I20240501 20:11:21.340445 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(75fa04c8726c4f2e932740cc6fbdcdd0): 4921 bytes on disk
I20240501 20:11:21.342149 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(4587a9d5392742b58ec8d64619f2dda1): perf score=0.033371
I20240501 20:11:21.347366 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(75fa04c8726c4f2e932740cc6fbdcdd0) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":3169,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240501 20:11:21.363662 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(ed9b47caf0a041fd962e4f18b0de5ff2): perf score=0.033369
I20240501 20:11:21.391316 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(75fa04c8726c4f2e932740cc6fbdcdd0): perf score=0.033390
I20240501 20:11:21.440075 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(4587a9d5392742b58ec8d64619f2dda1) complete. Timing: real 0.168s user 0.050s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":20361,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":54016,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1001,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":38352,"threads_started":2,"wal-append.queue_time_us":523}
I20240501 20:11:21.444667 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(4587a9d5392742b58ec8d64619f2dda1): 4581 bytes on disk
I20240501 20:11:21.446552 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(4587a9d5392742b58ec8d64619f2dda1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240501 20:11:21.449149 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(ed9b47caf0a041fd962e4f18b0de5ff2): perf score=0.033397
I20240501 20:11:21.467852 5351 ts_tablet_manager.cc:902] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:11:21.500810 5357 ts_tablet_manager.cc:902] T 6c3bdb74d3d140e397ce0b5bdef5361d P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
I20240501 20:11:21.526463 5351 tablet_copy_client.cc:308] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.136:44267
I20240501 20:11:21.528674 24130 tablet_copy_service.cc:140] P 8559229e044b4e1ea661b881b3f9af50: Received BeginTabletCopySession request for tablet 726a7cd006104059b4b10e6eab407577 from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:52271)
I20240501 20:11:21.528901 24130 tablet_copy_service.cc:161] P 8559229e044b4e1ea661b881b3f9af50: Beginning new tablet copy session on tablet 726a7cd006104059b4b10e6eab407577 from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:52271: session id = 61380023001a4ee9ada9ec2c6c1b7159-726a7cd006104059b4b10e6eab407577
I20240501 20:11:21.531862 24130 tablet_copy_source_session.cc:205] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:21.533177 5351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726a7cd006104059b4b10e6eab407577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:21.533188 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(75fa04c8726c4f2e932740cc6fbdcdd0) complete. Timing: real 0.142s user 0.033s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12232,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":5925,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":995,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":17510,"threads_started":2,"wal-append.queue_time_us":4991}
I20240501 20:11:21.536197 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(75fa04c8726c4f2e932740cc6fbdcdd0): 4921 bytes on disk
I20240501 20:11:21.540113 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(75fa04c8726c4f2e932740cc6fbdcdd0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240501 20:11:21.560040 5357 tablet_copy_client.cc:308] T 6c3bdb74d3d140e397ce0b5bdef5361d P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.130:42271
I20240501 20:11:21.573973 21118 tablet_copy_service.cc:140] P feac73db0a3b4aafa374ca5dfd322b08: Received BeginTabletCopySession request for tablet 6c3bdb74d3d140e397ce0b5bdef5361d from peer 4af096f5a6a9440a94f6682b399efe20 ({username='slave'} at 127.13.251.131:45933)
I20240501 20:11:21.574220 21118 tablet_copy_service.cc:161] P feac73db0a3b4aafa374ca5dfd322b08: Beginning new tablet copy session on tablet 6c3bdb74d3d140e397ce0b5bdef5361d from peer 4af096f5a6a9440a94f6682b399efe20 at {username='slave'} at 127.13.251.131:45933: session id = 4af096f5a6a9440a94f6682b399efe20-6c3bdb74d3d140e397ce0b5bdef5361d
I20240501 20:11:21.571496 5356 ts_tablet_manager.cc:902] T a1cbbb090cb34465b6dac9c35cdba0dc P 4af096f5a6a9440a94f6682b399efe20: Initiating tablet copy from peer feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271)
F20240501 20:11:21.575390 5356 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0722c64801734730abefe2a742da549e.queue_time_us, tablet-download-pool-0722c64801734730abefe2a742da549e.queue_time_us) (tablet-download-pool-0722c64801734730abefe2a742da549e.run_cpu_time_us, tablet-download-pool-0722c64801734730abefe2a742da549e.run_cpu_time_us) (tablet-download-pool-0722c64801734730abefe2a742da549e.run_wall_time_us, tablet-download-pool-0722c64801734730abefe2a742da549e.run_wall_time_us) (tablet-download-pool-224f83ad3d514a7fa5cdfa4e70ff2fd8.queue_time_us, tablet-download-pool-224f83ad3d514a7fa5cdfa4e70ff2fd8.queue_time_us) (tablet-download-pool-224f83ad3d514a7fa5cdfa4e70ff2fd8.run_cpu_time_us, tablet-download-pool-224f83ad3d514a7fa5cdfa4e70ff2fd8.run_cpu_time_us) (tablet-download-pool-224f83ad3d514a7fa5cdfa4e70ff2fd8.run_wall_time_us, tablet-download-pool-224f83ad3d514a7fa5cdfa4e70ff2fd8.run_wall_time_us) (tablet-download-pool-236ba60733e1416395346179be3b3888.queue_time_us, tablet-download-pool-236ba60733e1416395346179be3b3888.queue_time_us) (tablet-download-pool-236ba60733e1416395346179be3b3888.run_cpu_time_us, tablet-download-pool-236ba60733e1416395346179be3b3888.run_cpu_time_us) (tablet-download-pool-236ba60733e1416395346179be3b3888.run_wall_time_us, tablet-download-pool-236ba60733e1416395346179be3b3888.run_wall_time_us) (tablet-download-pool-2b57a80759944301a033140e482aec36.queue_time_us, tablet-download-pool-2b57a80759944301a033140e482aec36.queue_time_us) (tablet-download-pool-2b57a80759944301a033140e482aec36.run_cpu_time_us, tablet-download-pool-2b57a80759944301a033140e482aec36.run_cpu_time_us) (tablet-download-pool-2b57a80759944301a033140e482aec36.run_wall_time_us, tablet-download-pool-2b57a80759944301a033140e482aec36.run_wall_time_us) (tablet-download-pool-3f3951bb18524e41aa6836122e97e7f5.queue_time_us, tablet-download-pool-3f3951bb18524e41aa6836122e97e7f5.queue_time_us) (tablet-download-pool-3f3951bb18524e41aa6836122e97e7f5.run_cpu_time_us, tablet-download-pool-3f3951bb18524e41aa6836122e97e7f5.run_cpu_time_us) (tablet-download-pool-3f3951bb18524e41aa6836122e97e7f5.run_wall_time_us, tablet-download-pool-3f3951bb18524e41aa6836122e97e7f5.run_wall_time_us) (tablet-download-pool-4341f17784bc45359a9648ab47c39628.queue_time_us, tablet-download-pool-4341f17784bc45359a9648ab47c39628.queue_time_us) (tablet-download-pool-4341f17784bc45359a9648ab47c39628.run_cpu_time_us, tablet-download-pool-4341f17784bc45359a9648ab47c39628.run_cpu_time_us) (tablet-download-pool-4341f17784bc45359a9648ab47c39628.run_wall_time_us, tablet-download-pool-4341f17784bc45359a9648ab47c39628.run_wall_time_us) (tablet-download-pool-46677b6d96bf45d98d061e572a85ebdc.queue_time_us, tablet-download-pool-46677b6d96bf45d98d061e572a85ebdc.queue_time_us) (tablet-download-pool-46677b6d96bf45d98d061e572a85ebdc.run_cpu_time_us, tablet-download-pool-46677b6d96bf45d98d061e572a85ebdc.run_cpu_time_us) (tablet-download-pool-46677b6d96bf45d98d061e572a85ebdc.run_wall_time_us, tablet-download-pool-46677b6d96bf45d98d061e572a85ebdc.run_wall_time_us) (tablet-download-pool-4715a768081a49228ab47b8071c40c8f.queue_time_us, tablet-download-pool-4715a768081a49228ab47b8071c40c8f.queue_time_us) (tablet-download-pool-4715a768081a49228ab47b8071c40c8f.run_cpu_time_us, tablet-download-pool-4715a768081a49228ab47b8071c40c8f.run_cpu_time_us) (tablet-download-pool-4715a768081a49228ab47b8071c40c8f.run_wall_time_us, tablet-download-pool-4715a768081a49228ab47b8071c40c8f.run_wall_time_us) (tablet-download-pool-6c3bdb74d3d140e397ce0b5bdef5361d.queue_time_us, tablet-download-pool-6c3bdb74d3d140e397ce0b5bdef5361d.queue_time_us) (tablet-download-pool-6c3bdb74d3d140e397ce0b5bdef5361d.run_cpu_time_us, tablet-download-pool-6c3bdb74d3d140e397ce0b5bdef5361d.run_cpu_time_us) (tablet-download-pool-6c3bdb74d3d140e397ce0b5bdef5361d.run_wall_time_us, tablet-download-pool-6c3bdb74d3d140e397ce0b5bdef5361d.run_wall_time_us) (tablet-download-pool-7a23f60c87794a49ae1f82dff06fe881.queue_time_us, tablet-download-pool-7a23f60c87794a49ae1f82dff06fe881.queue_time_us) (tablet-download-pool-7a23f60c87794a49ae1f82dff06fe881.run_cpu_time_us, tablet-download-pool-7a23f60c87794a49ae1f82dff06fe881.run_cpu_time_us) (tablet-download-pool-7a23f60c87794a49ae1f82dff06fe881.run_wall_time_us, tablet-download-pool-7a23f60c87794a49ae1f82dff06fe881.run_wall_time_us) (tablet-download-pool-a1cbbb090cb34465b6dac9c35cdba0dc.queue_time_us, tablet-download-pool-a1cbbb090cb34465b6dac9c35cdba0dc.queue_time_us) (tablet-download-pool-a7498fc5cf5144d58712eb2c8f188c5e.queue_time_us, tablet-download-pool-a7498fc5cf5144d58712eb2c8f188c5e.queue_time_us) (tablet-download-pool-a7498fc5cf5144d58712eb2c8f188c5e.run_cpu_time_us, tablet-download-pool-a7498fc5cf5144d58712eb2c8f188c5e.run_cpu_time_us) (tablet-download-pool-a7498fc5cf5144d58712eb2c8f188c5e.run_wall_time_us, tablet-download-pool-a7498fc5cf5144d58712eb2c8f188c5e.run_wall_time_us) (tablet-download-pool-c018d1b99ad749e59a2cf338001d38e4.queue_time_us, tablet-download-pool-c018d1b99ad749e59a2cf338001d38e4.queue_time_us) (tablet-download-pool-c018d1b99ad749e59a2cf338001d38e4.run_cpu_time_us, tablet-download-pool-c018d1b99ad749e59a2cf338001d38e4.run_cpu_time_us) (tablet-download-pool-c018d1b99ad749e59a2cf338001d38e4.run_wall_time_us, tablet-download-pool-c018d1b99ad749e59a2cf338001d38e4.run_wall_time_us) (tablet-download-pool-cf1ac1f2f3d642e9a9c785bb4963af1f.queue_time_us, tablet-download-pool-cf1ac1f2f3d642e9a9c785bb4963af1f.queue_time_us) (tablet-download-pool-cf1ac1f2f3d642e9a9c785bb4963af1f.run_cpu_time_us, tablet-download-pool-cf1ac1f2f3d642e9a9c785bb4963af1f.run_cpu_time_us) (tablet-download-pool-cf1ac1f2f3d642e9a9c785bb4963af1f.run_wall_time_us, tablet-download-pool-cf1ac1f2f3d642e9a9c785bb4963af1f.run_wall_time_us) (tablet-download-pool-e1be0c0baa5e4c95915b93f659c4759f.queue_time_us, tablet-download-pool-e1be0c0baa5e4c95915b93f659c4759f.queue_time_us) (tablet-download-pool-e1be0c0baa5e4c95915b93f659c4759f.run_cpu_time_us, tablet-download-pool-e1be0c0baa5e4c95915b93f659c4759f.run_cpu_time_us) (tablet-download-pool-e1be0c0baa5e4c95915b93f659c4759f.run_wall_time_us, tablet-download-pool-e1be0c0baa5e4c95915b93f659c4759f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 20:11:21.580494 21118 tablet_copy_source_session.cc:205] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:21.583032 5357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3bdb74d3d140e397ce0b5bdef5361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:21.582998 5351 tablet_copy_client.cc:783] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:21.583431 5351 tablet_copy_client.cc:647] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
@ 0x7f62c1abfdcd google::LogMessage::Fail() at ??:0
@ 0x7f62c1ac3b93 google::LogMessage::SendToLog() at ??:0
I20240501 20:11:21.595240 5357 tablet_copy_client.cc:783] T 6c3bdb74d3d140e397ce0b5bdef5361d P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:21.595554 5357 tablet_copy_client.cc:647] T 6c3bdb74d3d140e397ce0b5bdef5361d P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:21.598440 5357 tablet_copy_client.cc:523] T 6c3bdb74d3d140e397ce0b5bdef5361d P 4af096f5a6a9440a94f6682b399efe20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f62c1abf7cc google::LogMessage::Flush() at ??:0
@ 0x7f62c1ac0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 20:11:21.609784 5357 tablet_bootstrap.cc:492] T 6c3bdb74d3d140e397ce0b5bdef5361d P 4af096f5a6a9440a94f6682b399efe20: Bootstrap starting.
@ 0x7f62c4800742 kudu::TraceMetrics::InternName() at ??:0
I20240501 20:11:21.614926 5351 tablet_copy_client.cc:523] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:21.618549 5351 tablet_bootstrap.cc:492] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
@ 0x7f62c47ccf8b kudu::ThreadPool::ThreadPool() at ??:0
I20240501 20:11:21.622371 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(ed9b47caf0a041fd962e4f18b0de5ff2) complete. Timing: real 0.258s user 0.050s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1713,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":944,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":79804,"threads_started":1,"wal-append.queue_time_us":10685}
@ 0x7f62c47c6cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 20:11:21.625506 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(ed9b47caf0a041fd962e4f18b0de5ff2): 4834 bytes on disk
I20240501 20:11:21.631033 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(ed9b47caf0a041fd962e4f18b0de5ff2) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
@ 0x7f62d3c198de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f62d3c31daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 20:11:21.641906 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(ed9b47caf0a041fd962e4f18b0de5ff2) complete. Timing: real 0.192s user 0.050s sys 0.044s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":656,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1559,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":857,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":83141,"threads_started":1,"wal-append.queue_time_us":82780}
I20240501 20:11:21.644922 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(ed9b47caf0a041fd962e4f18b0de5ff2): 4834 bytes on disk
I20240501 20:11:21.653664 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(ed9b47caf0a041fd962e4f18b0de5ff2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 20:11:21.656968 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: FlushMRSOp(4587a9d5392742b58ec8d64619f2dda1) complete. Timing: real 0.314s user 0.059s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8700,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":6700,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":949,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":21471,"threads_started":2,"wal-append.queue_time_us":2803}
I20240501 20:11:21.685580 5351 tablet_bootstrap.cc:492] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:21.685979 5351 tablet_bootstrap.cc:492] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:11:21.686213 5351 ts_tablet_manager.cc:1367] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.003s
I20240501 20:11:21.686612 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling UndoDeltaBlockGCOp(4587a9d5392742b58ec8d64619f2dda1): 4581 bytes on disk
I20240501 20:11:21.686956 5351 raft_consensus.cc:348] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:21.687173 5351 raft_consensus.cc:729] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:11:21.687439 5351 consensus_queue.cc:260] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:21.690789 5351 ts_tablet_manager.cc:1398] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:11:21.691403 24130 tablet_copy_service.cc:342] P 8559229e044b4e1ea661b881b3f9af50: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-726a7cd006104059b4b10e6eab407577 received from {username='slave'} at 127.13.251.132:52271
I20240501 20:11:21.691510 24130 tablet_copy_service.cc:433] P 8559229e044b4e1ea661b881b3f9af50: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-726a7cd006104059b4b10e6eab407577 on tablet 726a7cd006104059b4b10e6eab407577 with peer 61380023001a4ee9ada9ec2c6c1b7159
@ 0x7f62d3d5c480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 20:11:21.709810 21207 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
@ 0x7f62d3d94a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 20:11:21.720913 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:21.738786 21165 maintenance_manager.cc:643] P 4af096f5a6a9440a94f6682b399efe20: UndoDeltaBlockGCOp(4587a9d5392742b58ec8d64619f2dda1) complete. Timing: real 0.012s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20240501 20:11:21.744657 21274 maintenance_manager.cc:419] P 4af096f5a6a9440a94f6682b399efe20: Scheduling FlushMRSOp(ed9b47caf0a041fd962e4f18b0de5ff2): perf score=0.033478
@ 0x7f62d3d77981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f62d3d7770e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f62c47d0b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f62c47d419c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f62c47d4019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f62c47af01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f62c7b346db start_thread at ??:0
@ 0x7f62c097271f clone at ??:0
I20240501 20:11:21.833954 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:21.899140 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:21.935807 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:21.960171 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:22.037609 21358 raft_consensus.cc:1203] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:11:22.037859 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:22.059916 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:22.280290 23647 connection.cc:535] client connection to 127.13.251.131:38349 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.280588 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.281026 23782 connection.cc:535] client connection to 127.13.251.131:38349 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.281140 24046 connection.cc:535] client connection to 127.13.251.131:38349 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.281250 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.281349 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.281780 21294 connection.cc:535] client connection to 127.13.251.131:38349 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.282042 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.284176 20903 connection.cc:535] client connection to 127.13.251.131:38349 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.284545 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:22.285696 21034 connection.cc:698] client connection to 127.13.251.131:38349 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20240501 20:11:22.285907 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: sendmsg error: Connection reset by peer (error 104)
W20240501 20:11:22.306335 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.306558 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.306666 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.306761 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.307168 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.307278 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.307348 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.307415 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.307498 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:22.307642 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.307740 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.312047 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.312319 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.312485 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.312647 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.312798 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.312949 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.313105 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.318488 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.320067 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.320206 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.320269 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.324327 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.325615 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.325738 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.325893 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.325975 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.326052 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.326136 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.326212 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.326289 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.366914 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:22.417420 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.674502 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:22.680714 24089 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:22.682240 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:22.701714 21338 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:22.751149 5310 raft_consensus.cc:482] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.751350 5310 raft_consensus.cc:504] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:22.752202 5310 leader_election.cc:288] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:22.760780 23842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b472171e77e144ffa8fa4cdbd40a03a0" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:11:22.760828 23825 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:22.764098 20903 leader_election.cc:334] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.764247 20903 leader_election.cc:302] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0; no voters: 4af096f5a6a9440a94f6682b399efe20, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:11:22.764511 4979 raft_consensus.cc:2726] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:22.780751 4879 raft_consensus.cc:482] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.780917 4879 raft_consensus.cc:504] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:22.781736 4879 leader_election.cc:288] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:11:22.782322 20962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538f2f3892624f63bc88064e5ecd9443" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:11:22.783071 20945 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240501 20:11:22.784343 23647 leader_election.cc:334] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.784459 23647 leader_election.cc:302] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46; no voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0
I20240501 20:11:22.784931 4879 raft_consensus.cc:2726] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:22.797921 4979 raft_consensus.cc:482] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.798094 4979 raft_consensus.cc:504] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:22.798810 4979 leader_election.cc:288] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
W20240501 20:11:22.801805 20903 leader_election.cc:334] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.803663 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718853c0da3d4a68ab478a6e79cdd438" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:11:22.803934 21077 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:22.822490 20902 leader_election.cc:302] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cda19d6a81c0473db302535ce2ada7f0; no voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:11:22.823086 4979 raft_consensus.cc:2726] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:22.847363 4974 raft_consensus.cc:482] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.851394 4974 raft_consensus.cc:504] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:22.852206 4974 leader_election.cc:288] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:22.854863 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ccc5b4ec949f1aa0652e43a3b9e5c" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
W20240501 20:11:22.854799 24046 leader_election.cc:334] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.862771 24043 leader_election.cc:302] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:11:22.864398 5492 raft_consensus.cc:2726] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:22.880533 5345 raft_consensus.cc:482] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.880726 5345 raft_consensus.cc:504] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
W20240501 20:11:22.881145 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.881556 5345 leader_election.cc:288] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:11:22.882660 20962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b472171e77e144ffa8fa4cdbd40a03a0" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:11:22.882869 20962 raft_consensus.cc:2454] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:11:22.883275 23779 leader_election.cc:302] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, cda19d6a81c0473db302535ce2ada7f0; no voters:
W20240501 20:11:22.883562 23782 leader_election.cc:334] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.883601 5345 raft_consensus.cc:2781] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:22.883709 5345 raft_consensus.cc:482] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.883802 5345 raft_consensus.cc:3037] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:22.889149 5345 raft_consensus.cc:504] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:22.890179 20962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b472171e77e144ffa8fa4cdbd40a03a0" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:11:22.890359 20962 raft_consensus.cc:3037] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:11:22.891252 23782 leader_election.cc:334] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.891420 5345 leader_election.cc:288] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:11:22.892903 5461 raft_consensus.cc:482] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.893105 5461 raft_consensus.cc:504] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:11:22.893299 20962 raft_consensus.cc:2454] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 2.
I20240501 20:11:22.894167 5461 leader_election.cc:288] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:11:22.894557 23779 leader_election.cc:302] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, cda19d6a81c0473db302535ce2ada7f0; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:22.894835 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:11:22.895253 5345 raft_consensus.cc:2781] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:22.895401 5345 raft_consensus.cc:686] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:11:22.895686 5345 consensus_queue.cc:237] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:22.900957 20820 catalog_manager.cc:5484] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 2 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
W20240501 20:11:22.916713 23912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:22.920774 23912 leader_election.cc:334] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.920922 23912 leader_election.cc:302] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb; no voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0
I20240501 20:11:22.921332 5461 raft_consensus.cc:2726] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:11:22.928301 5461 raft_consensus.cc:482] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.928493 5461 raft_consensus.cc:504] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:11:22.929240 5461 leader_election.cc:288] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:22.929678 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e27bcc1cf34d8ab5d50331bee12886" candidate_uuid: "744c5a532e724391830c0785d32eb2fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
W20240501 20:11:22.931041 23912 leader_election.cc:334] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.931140 23912 leader_election.cc:302] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb; no voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:11:22.931491 5461 raft_consensus.cc:2726] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:22.941890 24107 consensus_queue.cc:237] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:22.944201 21358 raft_consensus.cc:1261] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:22.944272 23710 raft_consensus.cc:1261] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:22.944907 5492 consensus_queue.cc:1035] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:22.945264 4974 consensus_queue.cc:1035] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:22.948256 5492 raft_consensus.cc:2932] T 726a7cd006104059b4b10e6eab407577 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:22.948642 21358 raft_consensus.cc:2932] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:22.956427 20820 catalog_manager.cc:5484] T 726a7cd006104059b4b10e6eab407577 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:22.960868 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 726a7cd006104059b4b10e6eab407577 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:11:22.965409 4982 raft_consensus.cc:482] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.965557 4982 raft_consensus.cc:504] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:22.966252 4982 leader_election.cc:288] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
W20240501 20:11:22.968140 23647 leader_election.cc:334] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.979897 5345 raft_consensus.cc:482] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.979892 4888 raft_consensus.cc:482] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.980106 5345 raft_consensus.cc:504] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:22.980111 4888 raft_consensus.cc:504] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:22.980813 4888 leader_election.cc:288] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259)
I20240501 20:11:22.980855 5345 leader_election.cc:288] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:22.981827 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d75b25758954877ab36e8029fb29463" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:11:22.982268 20965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4d24c5c844c8c9ccd191b1a2b0b00" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
W20240501 20:11:22.982726 23782 leader_election.cc:334] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:22.982857 23782 leader_election.cc:334] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.983132 23779 leader_election.cc:302] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:11:22.983516 23779 leader_election.cc:302] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20, cda19d6a81c0473db302535ce2ada7f0
I20240501 20:11:22.983752 4888 raft_consensus.cc:2726] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:22.983911 4888 raft_consensus.cc:2726] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:22.984315 5098 raft_consensus.cc:482] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.984500 5098 raft_consensus.cc:504] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:22.985494 5098 leader_election.cc:288] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:11:22.985954 5521 tablet_replica.cc:330] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:11:22.986168 5521 raft_consensus.cc:2229] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:22.986397 5521 raft_consensus.cc:2258] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:22.987092 5492 raft_consensus.cc:482] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.986953 23710 raft_consensus.cc:2932] T 726a7cd006104059b4b10e6eab407577 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:22.987241 5492 raft_consensus.cc:504] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:22.988193 5492 leader_election.cc:288] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:11:22.988314 5492 raft_consensus.cc:482] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.988433 5492 raft_consensus.cc:504] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:22.988994 5492 leader_election.cc:288] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:11:22.989709 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:11:22.989763 5521 ts_tablet_manager.cc:1875] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:22.989888 24109 raft_consensus.cc:2454] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 1.
I20240501 20:11:22.989842 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e295a46a174f3f8673e03f57890b58" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:11:22.990025 21358 raft_consensus.cc:2454] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:11:22.990578 24044 leader_election.cc:302] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:11:22.990892 23646 leader_election.cc:302] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:22.991362 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e295a46a174f3f8673e03f57890b58" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:11:22.991470 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:11:22.991627 24109 raft_consensus.cc:2454] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:11:22.991672 23709 raft_consensus.cc:2454] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:11:22.992863 21293 leader_election.cc:302] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters:
W20240501 20:11:22.993398 24046 leader_election.cc:334] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.993423 4814 raft_consensus.cc:2781] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:22.993501 4814 raft_consensus.cc:482] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
W20240501 20:11:22.993530 24046 leader_election.cc:334] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.993589 4814 raft_consensus.cc:3037] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:22.995034 4879 raft_consensus.cc:2781] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:22.995106 4879 raft_consensus.cc:482] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:22.995169 4879 raft_consensus.cc:3037] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:11:22.996632 21294 leader_election.cc:334] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:22.998898 5521 ts_tablet_manager.cc:1888] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:22.999083 5521 log.cc:1196] T 726a7cd006104059b4b10e6eab407577 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/726a7cd006104059b4b10e6eab407577
I20240501 20:11:22.998960 4879 raft_consensus.cc:504] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:23.000139 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet 726a7cd006104059b4b10e6eab407577 (table rebalancer_extra_table_28 [id=7e6a9f27b1944ae68de38b424ff1680e]) successfully deleted
I20240501 20:11:22.999912 4879 leader_election.cc:288] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
W20240501 20:11:23.001623 23647 leader_election.cc:334] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.001595 4814 raft_consensus.cc:504] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.003082 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e295a46a174f3f8673e03f57890b58" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:11:23.003350 24109 raft_consensus.cc:3037] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.006023 24045 leader_election.cc:302] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.006279 5492 raft_consensus.cc:2781] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.006398 5492 raft_consensus.cc:482] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.006515 5492 raft_consensus.cc:3037] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.008558 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
W20240501 20:11:23.008781 21294 leader_election.cc:334] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.009989 24109 raft_consensus.cc:2454] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 2.
I20240501 20:11:23.010231 4814 leader_election.cc:288] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:11:23.010422 5519 raft_consensus.cc:2741] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240501 20:11:23.010470 21293 leader_election.cc:302] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.009485 5492 raft_consensus.cc:504] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:23.010838 24107 raft_consensus.cc:2379] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bf640e895b4e5eb91bdf7509ea7e46 in current term 2: Already voted for candidate 8559229e044b4e1ea661b881b3f9af50 in this term.
I20240501 20:11:23.011188 4814 raft_consensus.cc:2781] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.011502 4814 raft_consensus.cc:686] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:11:23.011860 23646 leader_election.cc:302] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46; no voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:11:23.012336 5523 raft_consensus.cc:2726] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:11:23.012797 5492 leader_election.cc:288] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:11:23.011871 4814 consensus_queue.cc:237] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.013342 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
W20240501 20:11:23.013494 24046 leader_election.cc:334] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.013535 5523 raft_consensus.cc:482] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.013731 23710 raft_consensus.cc:2379] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8559229e044b4e1ea661b881b3f9af50 in current term 2: Already voted for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in this term.
I20240501 20:11:23.014228 24045 leader_election.cc:302] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20, 51bf640e895b4e5eb91bdf7509ea7e46
I20240501 20:11:23.014600 5492 raft_consensus.cc:2726] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:11:23.013643 5523 raft_consensus.cc:504] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:23.016849 5523 leader_election.cc:288] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:23.017882 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc6ff1a0104ac690c8d8f82c17200c" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
W20240501 20:11:23.019850 23647 leader_election.cc:334] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.019953 23647 leader_election.cc:302] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46; no voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:11:23.021164 5523 raft_consensus.cc:2726] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:23.021767 4814 raft_consensus.cc:482] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.021818 20820 catalog_manager.cc:5484] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 2 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:23.021884 4814 raft_consensus.cc:504] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:11:23.022989 23975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e27bcc1cf34d8ab5d50331bee12886" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:11:23.023164 23975 raft_consensus.cc:2454] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:11:23.023571 21291 leader_election.cc:302] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:11:23.023797 5098 raft_consensus.cc:2781] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.023854 5098 raft_consensus.cc:482] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.023907 5098 raft_consensus.cc:3037] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:11:23.025120 21294 leader_election.cc:334] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.026602 5098 raft_consensus.cc:504] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:11:23.027113 5098 leader_election.cc:288] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:11:23.028453 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e27bcc1cf34d8ab5d50331bee12886" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:11:23.028628 23977 raft_consensus.cc:3037] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.031370 23977 raft_consensus.cc:2454] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 2.
I20240501 20:11:23.033408 21291 leader_election.cc:302] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 744c5a532e724391830c0785d32eb2fb; no voters:
I20240501 20:11:23.036307 5098 raft_consensus.cc:2781] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.036592 5098 raft_consensus.cc:686] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:11:23.036979 5098 consensus_queue.cc:237] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:11:23.038241 4814 leader_election.cc:288] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:11:23.045532 4814 raft_consensus.cc:482] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.045780 4814 raft_consensus.cc:504] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
W20240501 20:11:23.045583 21294 leader_election.cc:334] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.046617 4814 leader_election.cc:288] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
W20240501 20:11:23.048475 21294 leader_election.cc:334] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.049511 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc6ff1a0104ac690c8d8f82c17200c" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:11:23.049746 23710 raft_consensus.cc:2454] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:11:23.050166 21293 leader_election.cc:302] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 61380023001a4ee9ada9ec2c6c1b7159; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.050437 5529 raft_consensus.cc:2781] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.050525 5529 raft_consensus.cc:482] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.050647 5529 raft_consensus.cc:3037] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.053972 20820 catalog_manager.cc:5484] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 2 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:11:23.054206 5529 raft_consensus.cc:504] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
W20240501 20:11:23.057004 21294 leader_election.cc:334] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.057042 5492 raft_consensus.cc:482] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.057193 5492 raft_consensus.cc:504] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.057808 5492 leader_election.cc:288] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:23.059139 5529 leader_election.cc:288] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:11:23.060210 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc6ff1a0104ac690c8d8f82c17200c" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:11:23.060412 23710 raft_consensus.cc:3037] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.060921 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e80df9af0e40c5acc4dea95d7e95a8" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
W20240501 20:11:23.061846 24046 leader_election.cc:334] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.061965 24046 leader_election.cc:302] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20, 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:11:23.062881 23710 raft_consensus.cc:2454] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 2.
I20240501 20:11:23.063799 5492 raft_consensus.cc:2726] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:23.064194 21293 leader_election.cc:302] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 61380023001a4ee9ada9ec2c6c1b7159; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.065500 5530 raft_consensus.cc:2781] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.065623 5530 raft_consensus.cc:686] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:11:23.065857 5530 consensus_queue.cc:237] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:23.081753 5310 raft_consensus.cc:482] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.081980 5310 raft_consensus.cc:504] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:11:23.082928 5310 leader_election.cc:288] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:11:23.083413 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb" is_pre_election: true
I20240501 20:11:23.083645 23977 raft_consensus.cc:2454] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 2.
I20240501 20:11:23.084163 20900 leader_election.cc:302] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters:
W20240501 20:11:23.085254 20903 leader_election.cc:334] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.095782 5310 raft_consensus.cc:2781] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:11:23.095887 5310 raft_consensus.cc:482] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.095954 5310 raft_consensus.cc:3037] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:23.098845 5310 raft_consensus.cc:504] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
W20240501 20:11:23.101066 20903 leader_election.cc:334] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.102044 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2aada073b4d8989d7db8be4034756" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "744c5a532e724391830c0785d32eb2fb"
I20240501 20:11:23.102263 23977 raft_consensus.cc:3037] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:23.104931 23977 raft_consensus.cc:2454] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 3.
I20240501 20:11:23.105525 20900 leader_election.cc:302] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 744c5a532e724391830c0785d32eb2fb, cda19d6a81c0473db302535ce2ada7f0; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.105808 4979 raft_consensus.cc:2781] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:23.121302 5310 leader_election.cc:288] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 3 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 744c5a532e724391830c0785d32eb2fb (127.13.251.135:41019)
I20240501 20:11:23.127121 4979 raft_consensus.cc:686] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [term 3 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:11:23.127516 4979 consensus_queue.cc:237] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } }
I20240501 20:11:23.136369 5529 raft_consensus.cc:482] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.136577 5529 raft_consensus.cc:504] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.138175 5529 leader_election.cc:288] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:23.140641 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420b46edf5fc438fb01e822440bce8f1" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:11:23.141386 21294 leader_election.cc:334] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.141840 21294 leader_election.cc:302] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159; no voters: 4af096f5a6a9440a94f6682b399efe20, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:11:23.143505 5539 raft_consensus.cc:2726] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:23.143400 20820 catalog_manager.cc:5484] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 2 to 3, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 3 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } }
I20240501 20:11:23.144306 5213 raft_consensus.cc:482] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
W20240501 20:11:23.144346 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 76596.6us
I20240501 20:11:23.144464 5213 raft_consensus.cc:504] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.145095 5213 leader_election.cc:288] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:23.148015 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021fed14d5a140f3b42d359bae264f69" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
W20240501 20:11:23.149643 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 82708.2us
I20240501 20:11:23.151304 4814 raft_consensus.cc:482] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.151512 4814 raft_consensus.cc:504] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.152206 4814 leader_election.cc:288] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:11:23.154026 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e80df9af0e40c5acc4dea95d7e95a8" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:11:23.154247 24107 raft_consensus.cc:2454] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 1.
I20240501 20:11:23.155272 5541 raft_consensus.cc:482] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.155436 5541 raft_consensus.cc:504] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:23.155700 20820 catalog_manager.cc:5484] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 2 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:23.156038 5541 leader_election.cc:288] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
W20240501 20:11:23.159276 21034 leader_election.cc:334] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.159415 21034 leader_election.cc:302] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feac73db0a3b4aafa374ca5dfd322b08; no voters: 4af096f5a6a9440a94f6682b399efe20, c6dcaa8efff642dcb7c2c6e8be3a65f7
I20240501 20:11:23.159996 5546 raft_consensus.cc:2726] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:23.161803 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:11:23.162530 21293 leader_election.cc:302] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:11:23.163391 4814 raft_consensus.cc:2781] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.163487 4814 raft_consensus.cc:482] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.163591 4814 raft_consensus.cc:3037] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:11:23.163980 21294 leader_election.cc:334] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:23.165113 21294 leader_election.cc:334] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.166144 5345 raft_consensus.cc:482] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.166285 4814 raft_consensus.cc:504] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.166287 5345 raft_consensus.cc:504] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.166818 4814 leader_election.cc:288] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:11:23.166944 5345 leader_election.cc:288] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:11:23.168699 21293 leader_election.cc:302] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159; no voters: 4af096f5a6a9440a94f6682b399efe20, 8559229e044b4e1ea661b881b3f9af50
I20240501 20:11:23.169011 5541 raft_consensus.cc:2726] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:11:23.169531 21294 leader_election.cc:334] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.169898 5544 raft_consensus.cc:482] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.170085 5544 raft_consensus.cc:504] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.170591 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e80df9af0e40c5acc4dea95d7e95a8" candidate_uuid: "61380023001a4ee9ada9ec2c6c1b7159" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:11:23.170727 5544 leader_election.cc:288] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:23.170799 24107 raft_consensus.cc:3037] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:11:23.170986 23782 leader_election.cc:334] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.172793 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4d24c5c844c8c9ccd191b1a2b0b00" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:11:23.172982 23843 raft_consensus.cc:2454] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:11:23.173641 24107 raft_consensus.cc:2454] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61380023001a4ee9ada9ec2c6c1b7159 in term 2.
I20240501 20:11:23.174314 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420b46edf5fc438fb01e822440bce8f1" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:11:23.174528 21358 raft_consensus.cc:2454] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
I20240501 20:11:23.174839 23780 leader_election.cc:302] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.175089 5345 raft_consensus.cc:2781] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.175182 5345 raft_consensus.cc:482] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.175267 5345 raft_consensus.cc:3037] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.175350 21293 leader_election.cc:302] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.175438 20903 leader_election.cc:302] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:11:23.175621 5541 raft_consensus.cc:2781] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 20:11:23.175670 20903 leader_election.cc:334] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.175740 5541 raft_consensus.cc:686] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 LEADER]: Becoming Leader. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Running, Role: LEADER
I20240501 20:11:23.175992 5542 raft_consensus.cc:2781] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.175974 5541 consensus_queue.cc:237] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.176124 5542 raft_consensus.cc:482] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.176301 5542 raft_consensus.cc:3037] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.179288 5345 raft_consensus.cc:504] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.181540 5345 leader_election.cc:288] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
W20240501 20:11:23.181962 23782 leader_election.cc:334] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.184615 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420b46edf5fc438fb01e822440bce8f1" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:11:23.184881 21358 raft_consensus.cc:3037] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.184841 5542 raft_consensus.cc:504] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.188609 21358 raft_consensus.cc:2454] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 2.
I20240501 20:11:23.189908 23780 leader_election.cc:302] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, c6dcaa8efff642dcb7c2c6e8be3a65f7; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.190527 5345 raft_consensus.cc:2781] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.190763 5345 raft_consensus.cc:686] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:11:23.191025 5345 consensus_queue.cc:237] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
W20240501 20:11:23.191408 20903 leader_election.cc:334] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.192232 5542 leader_election.cc:288] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:23.192781 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4d24c5c844c8c9ccd191b1a2b0b00" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:11:23.192960 23843 raft_consensus.cc:3037] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.198650 20820 catalog_manager.cc:5484] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 2 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:23.201375 20822 catalog_manager.cc:5484] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132). New cstate: current_term: 2 leader_uuid: "61380023001a4ee9ada9ec2c6c1b7159" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:23.217018 23843 raft_consensus.cc:2454] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 2.
I20240501 20:11:23.217633 20903 leader_election.cc:302] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, cda19d6a81c0473db302535ce2ada7f0; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.217957 5542 raft_consensus.cc:2781] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.218111 5542 raft_consensus.cc:686] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [term 2 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:11:23.218407 5542 consensus_queue.cc:237] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.220253 5546 raft_consensus.cc:482] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.220410 5546 raft_consensus.cc:504] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.221047 5546 leader_election.cc:288] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
W20240501 20:11:23.222368 21034 leader_election.cc:334] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.222777 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ccc5b4ec949f1aa0652e43a3b9e5c" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:11:23.222994 24107 raft_consensus.cc:2454] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:11:23.223790 21033 leader_election.cc:302] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.224572 5546 raft_consensus.cc:2781] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.224671 5546 raft_consensus.cc:482] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.224761 5546 raft_consensus.cc:3037] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.227523 5546 raft_consensus.cc:504] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
W20240501 20:11:23.230057 21034 leader_election.cc:334] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.232982 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ccc5b4ec949f1aa0652e43a3b9e5c" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:11:23.233186 24107 raft_consensus.cc:3037] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.235739 20822 catalog_manager.cc:5484] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 2 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:23.244946 5492 raft_consensus.cc:482] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.245131 5492 raft_consensus.cc:504] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:23.245607 24107 raft_consensus.cc:2454] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:11:23.245837 5492 leader_election.cc:288] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:11:23.246143 21033 leader_election.cc:302] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, feac73db0a3b4aafa374ca5dfd322b08; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.246459 5213 raft_consensus.cc:2781] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.246580 5213 raft_consensus.cc:686] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [term 2 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:11:23.246816 5213 consensus_queue.cc:237] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.247418 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159" is_pre_election: true
I20240501 20:11:23.247658 21358 raft_consensus.cc:2454] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 2.
I20240501 20:11:23.248160 24044 leader_election.cc:302] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:11:23.248746 5546 leader_election.cc:288] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
W20240501 20:11:23.250083 24046 leader_election.cc:334] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.250473 5492 raft_consensus.cc:2781] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:11:23.250584 5492 raft_consensus.cc:482] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.250685 5492 raft_consensus.cc:3037] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:23.252837 4982 raft_consensus.cc:482] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.252990 4982 raft_consensus.cc:504] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:23.253644 4982 leader_election.cc:288] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:23.254026 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:11:23.253500 5492 raft_consensus.cc:504] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:23.254649 20822 catalog_manager.cc:5484] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 2 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
W20240501 20:11:23.255252 23647 leader_election.cc:334] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.255347 23647 leader_election.cc:302] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46; no voters: 4af096f5a6a9440a94f6682b399efe20, feac73db0a3b4aafa374ca5dfd322b08
I20240501 20:11:23.255595 4982 raft_consensus.cc:2726] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:23.255750 5492 leader_election.cc:288] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 3 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399)
I20240501 20:11:23.256078 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed53558d93214d34aedca8505870ec61" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61380023001a4ee9ada9ec2c6c1b7159"
I20240501 20:11:23.256294 21358 raft_consensus.cc:3037] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:23.258153 5345 raft_consensus.cc:482] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.258335 5345 raft_consensus.cc:504] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.259092 5345 leader_election.cc:288] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:23.259511 5213 raft_consensus.cc:482] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.259732 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021fed14d5a140f3b42d359bae264f69" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08" is_pre_election: true
I20240501 20:11:23.259933 21097 raft_consensus.cc:2454] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 1.
W20240501 20:11:23.259923 24046 leader_election.cc:334] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.260380 23779 leader_election.cc:302] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:11:23.260622 5345 raft_consensus.cc:2781] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.260701 5345 raft_consensus.cc:482] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.259055 21358 raft_consensus.cc:2454] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 3.
I20240501 20:11:23.260776 5345 raft_consensus.cc:3037] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:11:23.264464 23782 leader_election.cc:334] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.265064 5345 raft_consensus.cc:504] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.259693 5213 raft_consensus.cc:504] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.267899 5213 leader_election.cc:288] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:23.269392 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d75b25758954877ab36e8029fb29463" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" is_pre_election: true
I20240501 20:11:23.269615 23843 raft_consensus.cc:2454] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:11:23.270378 21034 leader_election.cc:302] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters:
W20240501 20:11:23.270678 21034 leader_election.cc:334] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.270817 5213 raft_consensus.cc:2781] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.270902 5213 raft_consensus.cc:482] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.270987 5213 raft_consensus.cc:3037] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.273238 21097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021fed14d5a140f3b42d359bae264f69" candidate_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "feac73db0a3b4aafa374ca5dfd322b08"
I20240501 20:11:23.273458 21097 raft_consensus.cc:3037] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.275331 5213 raft_consensus.cc:504] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.277379 5213 leader_election.cc:288] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431)
I20240501 20:11:23.278113 23843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d75b25758954877ab36e8029fb29463" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7"
I20240501 20:11:23.278312 23843 raft_consensus.cc:3037] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:11:23.278342 21034 leader_election.cc:334] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.279852 21097 raft_consensus.cc:2454] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dcaa8efff642dcb7c2c6e8be3a65f7 in term 2.
I20240501 20:11:23.280411 4807 raft_consensus.cc:482] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.280550 4807 raft_consensus.cc:504] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:23.280946 23843 raft_consensus.cc:2454] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:11:23.281196 4807 leader_election.cc:288] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:11:23.281656 5542 raft_consensus.cc:482] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.281860 5542 raft_consensus.cc:504] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:23.281991 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:11:23.282177 23710 raft_consensus.cc:2454] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 1.
I20240501 20:11:23.282543 21033 leader_election.cc:302] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:11:23.282680 5542 leader_election.cc:288] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:11:23.283097 4807 raft_consensus.cc:2781] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.283198 4807 raft_consensus.cc:482] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.283282 4807 raft_consensus.cc:3037] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.284250 21034 leader_election.cc:302] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters: 4af096f5a6a9440a94f6682b399efe20
W20240501 20:11:23.285606 21034 leader_election.cc:334] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.286367 4807 raft_consensus.cc:504] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:23.287031 4807 leader_election.cc:288] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
I20240501 20:11:23.287220 4807 raft_consensus.cc:2781] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.287331 4807 raft_consensus.cc:686] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:11:23.287760 23779 leader_election.cc:302] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6dcaa8efff642dcb7c2c6e8be3a65f7, feac73db0a3b4aafa374ca5dfd322b08; no voters:
I20240501 20:11:23.288379 5345 leader_election.cc:288] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 election: Requested vote from peers feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130:42271), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:23.288571 5345 raft_consensus.cc:2781] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.288702 5345 raft_consensus.cc:686] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 LEADER]: Becoming Leader. State: Replica: c6dcaa8efff642dcb7c2c6e8be3a65f7, State: Running, Role: LEADER
I20240501 20:11:23.288837 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d2ef39a5347e2b6fbe3145c207280" candidate_uuid: "feac73db0a3b4aafa374ca5dfd322b08" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
W20240501 20:11:23.288915 20903 leader_election.cc:334] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.289050 23709 raft_consensus.cc:3037] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.288980 5345 consensus_queue.cc:237] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.289314 24044 leader_election.cc:302] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61380023001a4ee9ada9ec2c6c1b7159, 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20
W20240501 20:11:23.289825 21034 leader_election.cc:334] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.289983 5492 raft_consensus.cc:2781] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:23.290011 4807 consensus_queue.cc:237] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } }
I20240501 20:11:23.291872 5520 raft_consensus.cc:482] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.292012 5520 raft_consensus.cc:504] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
W20240501 20:11:23.292212 23782 leader_election.cc:334] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.292558 5520 leader_election.cc:288] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:23.295579 20962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718853c0da3d4a68ab478a6e79cdd438" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0" is_pre_election: true
I20240501 20:11:23.295866 20962 raft_consensus.cc:2454] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 1.
I20240501 20:11:23.295831 20818 catalog_manager.cc:5484] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134). New cstate: current_term: 2 leader_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
I20240501 20:11:23.296321 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538f2f3892624f63bc88064e5ecd9443" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" is_pre_election: true
I20240501 20:11:23.296483 24043 leader_election.cc:302] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:11:23.296595 23710 raft_consensus.cc:2454] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 1.
I20240501 20:11:23.296722 5520 raft_consensus.cc:2781] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.296789 5520 raft_consensus.cc:482] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.296855 5520 raft_consensus.cc:3037] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.297192 20902 leader_election.cc:302] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, cda19d6a81c0473db302535ce2ada7f0; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.297729 5542 raft_consensus.cc:2781] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:23.297837 5542 raft_consensus.cc:482] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4af096f5a6a9440a94f6682b399efe20)
I20240501 20:11:23.297946 5542 raft_consensus.cc:3037] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.299389 5520 raft_consensus.cc:504] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.299979 5520 leader_election.cc:288] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: Requested vote from peers cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129:41259), 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349)
I20240501 20:11:23.301685 20962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718853c0da3d4a68ab478a6e79cdd438" candidate_uuid: "8559229e044b4e1ea661b881b3f9af50" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cda19d6a81c0473db302535ce2ada7f0"
I20240501 20:11:23.301906 20962 raft_consensus.cc:3037] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.302109 23709 raft_consensus.cc:2454] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feac73db0a3b4aafa374ca5dfd322b08 in term 2.
I20240501 20:11:23.301985 5542 raft_consensus.cc:504] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:23.305580 5492 raft_consensus.cc:686] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [term 3 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:11:23.305852 5492 consensus_queue.cc:237] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:23.306968 21033 leader_election.cc:302] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, feac73db0a3b4aafa374ca5dfd322b08; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.307749 5546 raft_consensus.cc:2781] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.307814 20962 raft_consensus.cc:2454] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8559229e044b4e1ea661b881b3f9af50 in term 2.
I20240501 20:11:23.307876 5546 raft_consensus.cc:686] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 LEADER]: Becoming Leader. State: Replica: feac73db0a3b4aafa374ca5dfd322b08, State: Running, Role: LEADER
I20240501 20:11:23.308184 5546 consensus_queue.cc:237] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:23.310060 24043 leader_election.cc:302] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8559229e044b4e1ea661b881b3f9af50, cda19d6a81c0473db302535ce2ada7f0; no voters:
I20240501 20:11:23.310540 5520 raft_consensus.cc:2781] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.311125 5520 raft_consensus.cc:686] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [term 2 LEADER]: Becoming Leader. State: Replica: 8559229e044b4e1ea661b881b3f9af50, State: Running, Role: LEADER
I20240501 20:11:23.311429 5520 consensus_queue.cc:237] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } }
I20240501 20:11:23.311976 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538f2f3892624f63bc88064e5ecd9443" candidate_uuid: "cda19d6a81c0473db302535ce2ada7f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "51bf640e895b4e5eb91bdf7509ea7e46"
I20240501 20:11:23.312196 23709 raft_consensus.cc:3037] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:23.313254 5542 leader_election.cc:288] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133:37155)
W20240501 20:11:23.313927 20903 leader_election.cc:334] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.315373 23709 raft_consensus.cc:2454] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda19d6a81c0473db302535ce2ada7f0 in term 2.
I20240501 20:11:23.318131 20902 leader_election.cc:302] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, cda19d6a81c0473db302535ce2ada7f0; no voters: 4af096f5a6a9440a94f6682b399efe20
I20240501 20:11:23.324754 5542 raft_consensus.cc:2781] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:23.328977 5542 raft_consensus.cc:686] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [term 2 LEADER]: Becoming Leader. State: Replica: cda19d6a81c0473db302535ce2ada7f0, State: Running, Role: LEADER
I20240501 20:11:23.329349 5542 consensus_queue.cc:237] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } }
I20240501 20:11:23.338384 20819 catalog_manager.cc:5484] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 2 to 3, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 3 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:23.341336 20822 catalog_manager.cc:5484] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 2 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } }
W20240501 20:11:23.341892 24046 leader_election.cc:334] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:23.345397 24046 leader_election.cc:334] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.348472 20818 catalog_manager.cc:5484] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to cda19d6a81c0473db302535ce2ada7f0 (127.13.251.129). New cstate: current_term: 2 leader_uuid: "cda19d6a81c0473db302535ce2ada7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } }
I20240501 20:11:23.351886 20820 catalog_manager.cc:5484] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136). New cstate: current_term: 2 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } }
I20240501 20:11:23.356140 20822 catalog_manager.cc:5484] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 reported cstate change: term changed from 1 to 2, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to feac73db0a3b4aafa374ca5dfd322b08 (127.13.251.130). New cstate: current_term: 2 leader_uuid: "feac73db0a3b4aafa374ca5dfd322b08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 } } peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:23.406992 24107 raft_consensus.cc:1261] T c1e295a46a174f3f8673e03f57890b58 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:23.407827 4814 consensus_queue.cc:1035] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:23.413204 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(998f71aeae984fd5b866cb866bd4e87a): perf score=0.033356
I20240501 20:11:23.438202 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(b4c4664b17f04e2d8285b7851b27d1a1): perf score=0.033363
W20240501 20:11:23.459856 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:23.475498 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:23.492923 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:23.513451 20962 raft_consensus.cc:1261] T b472171e77e144ffa8fa4cdbd40a03a0 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:23.514163 4888 consensus_queue.cc:1035] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:23.532102 23975 raft_consensus.cc:1261] T 30d2aada073b4d8989d7db8be4034756 P 744c5a532e724391830c0785d32eb2fb [term 3 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240501 20:11:23.532799 5544 consensus_queue.cc:1035] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:23.535559 23977 raft_consensus.cc:1261] T 62e27bcc1cf34d8ab5d50331bee12886 P 744c5a532e724391830c0785d32eb2fb [term 2 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:11:23.537689 4814 consensus_queue.cc:1035] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:23.564759 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(998f71aeae984fd5b866cb866bd4e87a): perf score=0.033398
W20240501 20:11:23.576243 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:23.577247 23709 raft_consensus.cc:1261] T 24cc6ff1a0104ac690c8d8f82c17200c P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:23.579263 5529 consensus_queue.cc:1035] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:11:23.586275 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:23.591404 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:23.613467 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:23.625561 23843 raft_consensus.cc:1261] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:23.628160 5544 consensus_queue.cc:1035] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:23.672597 24107 raft_consensus.cc:1261] T 170ccc5b4ec949f1aa0652e43a3b9e5c P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:11:23.675496 5546 consensus_queue.cc:1035] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:11:23.694136 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:23.697228 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:23.697651 20903 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 48879.9us
I20240501 20:11:23.700626 24109 raft_consensus.cc:1261] T 43e80df9af0e40c5acc4dea95d7e95a8 P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Refusing update from remote peer 61380023001a4ee9ada9ec2c6c1b7159: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:23.701738 5541 consensus_queue.cc:1035] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:11:23.712008 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:23.717715 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:23.738916 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 46716.3us
I20240501 20:11:23.745450 23842 raft_consensus.cc:1261] T 8d75b25758954877ab36e8029fb29463 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:23.746376 5546 consensus_queue.cc:1035] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:23.749131 23710 raft_consensus.cc:1261] T d75d2ef39a5347e2b6fbe3145c207280 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Refusing update from remote peer feac73db0a3b4aafa374ca5dfd322b08: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:23.750034 5574 consensus_queue.cc:1035] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:23.772171 21035 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.291s user 0.115s sys 0.130s
I20240501 20:11:23.776053 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(998f71aeae984fd5b866cb866bd4e87a) complete. Timing: real 0.363s user 0.160s sys 0.138s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6593,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":7282,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":799,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":297219,"threads_started":2,"wal-append.queue_time_us":290367}
I20240501 20:11:23.782217 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(998f71aeae984fd5b866cb866bd4e87a): 4834 bytes on disk
I20240501 20:11:23.783058 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(998f71aeae984fd5b866cb866bd4e87a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4}
I20240501 20:11:23.786839 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(b70e144824074927bb0f4eafcdeadcec): perf score=0.033458
W20240501 20:11:23.798494 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:23.799677 21358 raft_consensus.cc:1261] T 420b46edf5fc438fb01e822440bce8f1 P 61380023001a4ee9ada9ec2c6c1b7159 [term 2 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:23.800424 5573 consensus_queue.cc:1035] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:23.827451 23709 raft_consensus.cc:1261] T 538f2f3892624f63bc88064e5ecd9443 P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Refusing update from remote peer cda19d6a81c0473db302535ce2ada7f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:11:23.828320 5613 consensus_queue.cc:1035] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:23.849913 21097 raft_consensus.cc:1261] T 021fed14d5a140f3b42d359bae264f69 P feac73db0a3b4aafa374ca5dfd322b08 [term 2 FOLLOWER]: Refusing update from remote peer c6dcaa8efff642dcb7c2c6e8be3a65f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:23.850750 5573 consensus_queue.cc:1035] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feac73db0a3b4aafa374ca5dfd322b08" member_type: VOTER last_known_addr { host: "127.13.251.130" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:23.855525 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(998f71aeae984fd5b866cb866bd4e87a) complete. Timing: real 0.291s user 0.052s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5686,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":3655,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1045,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13764,"threads_started":2,"wal-append.queue_time_us":7753}
I20240501 20:11:23.859205 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(998f71aeae984fd5b866cb866bd4e87a): 4834 bytes on disk
I20240501 20:11:23.859863 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(998f71aeae984fd5b866cb866bd4e87a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
W20240501 20:11:23.875592 20903 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 155439us
I20240501 20:11:23.881631 20962 raft_consensus.cc:1261] T 718853c0da3d4a68ab478a6e79cdd438 P cda19d6a81c0473db302535ce2ada7f0 [term 2 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240501 20:11:23.884806 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:23.909971 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:23.910400 5492 consensus_queue.cc:1035] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda19d6a81c0473db302535ce2ada7f0" member_type: VOTER last_known_addr { host: "127.13.251.129" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:23.920750 24044 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 56260.7us
W20240501 20:11:23.923477 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:23.933415 21358 raft_consensus.cc:1261] T ed53558d93214d34aedca8505870ec61 P 61380023001a4ee9ada9ec2c6c1b7159 [term 3 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240501 20:11:23.938120 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:23.938309 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:23.940016 5520 consensus_queue.cc:1035] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:11:23.954643 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:23.963119 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(b4c4664b17f04e2d8285b7851b27d1a1) complete. Timing: real 0.525s user 0.050s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":17094,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":4808,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1062,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":112411,"threads_started":2,"wal-append.queue_time_us":94535}
I20240501 20:11:23.965867 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(b4c4664b17f04e2d8285b7851b27d1a1): 4921 bytes on disk
I20240501 20:11:23.966555 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(b4c4664b17f04e2d8285b7851b27d1a1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 20:11:24.045064 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(b70e144824074927bb0f4eafcdeadcec) complete. Timing: real 0.258s user 0.048s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":12742,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1746,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11936,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12204,"threads_started":1,"wal-append.queue_time_us":12108}
I20240501 20:11:24.053795 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(b70e144824074927bb0f4eafcdeadcec): 4581 bytes on disk
I20240501 20:11:24.054567 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(b70e144824074927bb0f4eafcdeadcec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 20:11:24.129659 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:24.146863 23691 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:24.172820 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:24.233688 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:24.242154 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.266125 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:24.271381 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:24.282769 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:24.291621 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:24.369808 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.395248 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:24.466187 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.482968 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.503080 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.523056 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.545616 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.554811 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.564121 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.575865 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.586987 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.591070 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.618266 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.653774 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.655155 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.660207 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.664419 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:24.675439 5523 raft_consensus.cc:482] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:24.675665 5523 raft_consensus.cc:504] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
I20240501 20:11:24.676275 5523 leader_election.cc:288] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
W20240501 20:11:24.677117 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:24.677152 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50" is_pre_election: true
I20240501 20:11:24.677383 24109 raft_consensus.cc:2454] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 2.
I20240501 20:11:24.677795 23646 leader_election.cc:302] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters:
I20240501 20:11:24.678064 5523 raft_consensus.cc:2781] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:11:24.678192 5523 raft_consensus.cc:482] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:24.678290 5523 raft_consensus.cc:3037] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:11:24.681771 23647 leader_election.cc:334] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:24.681031 5523 raft_consensus.cc:504] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
W20240501 20:11:24.696012 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:24.697556 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ffd0f0fb874a809876af2a78ad07ab" candidate_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8559229e044b4e1ea661b881b3f9af50"
I20240501 20:11:24.697804 24109 raft_consensus.cc:3037] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:11:24.700860 23647 leader_election.cc:334] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:24.701354 24109 raft_consensus.cc:2454] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bf640e895b4e5eb91bdf7509ea7e46 in term 3.
I20240501 20:11:24.701781 23646 leader_election.cc:302] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bf640e895b4e5eb91bdf7509ea7e46, 8559229e044b4e1ea661b881b3f9af50; no voters: 4af096f5a6a9440a94f6682b399efe20
W20240501 20:11:24.702065 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:24.702061 5653 raft_consensus.cc:2781] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:24.702245 5653 raft_consensus.cc:686] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [term 3 LEADER]: Becoming Leader. State: Replica: 51bf640e895b4e5eb91bdf7509ea7e46, State: Running, Role: LEADER
I20240501 20:11:24.702275 5523 leader_election.cc:288] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [CANDIDATE]: Term 3 election: Requested vote from peers 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349), 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:11:24.702536 5653 consensus_queue.cc:237] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } }
W20240501 20:11:24.730327 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:24.722249 20820 catalog_manager.cc:5484] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 reported cstate change: term changed from 1 to 3, leader changed from 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131) to 51bf640e895b4e5eb91bdf7509ea7e46 (127.13.251.133). New cstate: current_term: 3 leader_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af096f5a6a9440a94f6682b399efe20" member_type: VOTER last_known_addr { host: "127.13.251.131" port: 38349 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "51bf640e895b4e5eb91bdf7509ea7e46" member_type: VOTER last_known_addr { host: "127.13.251.133" port: 37155 } } }
I20240501 20:11:24.777060 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:24.784756 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:24.791666 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:24.793653 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.799525 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.803175 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.808156 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.821529 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.833317 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.833482 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.852573 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:24.861102 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:24.862347 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:24.864117 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:24.918526 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:24.956842 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:24.973445 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:25.152856 24109 consensus_queue.cc:237] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:25.155496 23842 raft_consensus.cc:1261] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:11:25.155519 24044 consensus_peers.cc:485] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132:46399): Couldn't send request to peer 61380023001a4ee9ada9ec2c6c1b7159. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7afaa4052754794b8f220501887af08. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:25.155872 23977 raft_consensus.cc:1261] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:25.156047 5492 consensus_queue.cc:1035] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:25.156383 5520 consensus_queue.cc:1035] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:11:25.160123 5520 raft_consensus.cc:2932] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:25.160552 23842 raft_consensus.cc:2932] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:25.166584 23977 raft_consensus.cc:2932] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:25.167714 20820 catalog_manager.cc:5484] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 reported cstate change: config changed from index -1 to 2, VOTER 61380023001a4ee9ada9ec2c6c1b7159 (127.13.251.132) added. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
W20240501 20:11:25.205456 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:25.208478 24107 raft_consensus.cc:1261] T d4ffd0f0fb874a809876af2a78ad07ab P 8559229e044b4e1ea661b881b3f9af50 [term 3 FOLLOWER]: Refusing update from remote peer 51bf640e895b4e5eb91bdf7509ea7e46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:11:25.209095 5653 consensus_queue.cc:1035] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:11:25.360139 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:25.436228 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:25.445533 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:25.475494 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:25.516088 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:25.519608 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:25.531740 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:25.537978 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:25.574349 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(4d14c3c71012405e81dabba5f127433d): perf score=0.033344
I20240501 20:11:25.637620 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(30d2aada073b4d8989d7db8be4034756): perf score=0.033361
I20240501 20:11:25.701767 5833 ts_tablet_manager.cc:902] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: Initiating tablet copy from peer 8559229e044b4e1ea661b881b3f9af50 (127.13.251.136:44267)
I20240501 20:11:25.712297 5833 tablet_copy_client.cc:308] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Beginning tablet copy session from remote peer at address 127.13.251.136:44267
I20240501 20:11:25.713299 24130 tablet_copy_service.cc:140] P 8559229e044b4e1ea661b881b3f9af50: Received BeginTabletCopySession request for tablet b7afaa4052754794b8f220501887af08 from peer 61380023001a4ee9ada9ec2c6c1b7159 ({username='slave'} at 127.13.251.132:52271)
I20240501 20:11:25.713474 24130 tablet_copy_service.cc:161] P 8559229e044b4e1ea661b881b3f9af50: Beginning new tablet copy session on tablet b7afaa4052754794b8f220501887af08 from peer 61380023001a4ee9ada9ec2c6c1b7159 at {username='slave'} at 127.13.251.132:52271: session id = 61380023001a4ee9ada9ec2c6c1b7159-b7afaa4052754794b8f220501887af08
I20240501 20:11:25.716431 24130 tablet_copy_source_session.cc:205] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:25.717845 5833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7afaa4052754794b8f220501887af08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:25.726536 5833 tablet_copy_client.cc:783] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:25.726964 5833 tablet_copy_client.cc:647] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:25.732385 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(4d14c3c71012405e81dabba5f127433d) complete. Timing: real 0.158s user 0.046s sys 0.037s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":40983,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":5121,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1000,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":48166,"threads_started":2,"wal-append.queue_time_us":7172}
I20240501 20:11:25.735095 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(4d14c3c71012405e81dabba5f127433d): 4921 bytes on disk
I20240501 20:11:25.737177 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(748b4fd500f84a359bd8ee05b861300d): perf score=0.033383
I20240501 20:11:25.737870 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(4d14c3c71012405e81dabba5f127433d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 20:11:25.741000 5833 tablet_copy_client.cc:523] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:25.745038 5833 tablet_bootstrap.cc:492] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap starting.
I20240501 20:11:25.793730 5833 tablet_bootstrap.cc:492] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:25.795842 5833 tablet_bootstrap.cc:492] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: Bootstrap complete.
I20240501 20:11:25.796423 5833 ts_tablet_manager.cc:1367] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.029s
I20240501 20:11:25.796360 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(30d2aada073b4d8989d7db8be4034756) complete. Timing: real 0.158s user 0.064s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":24919,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":6204,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":821,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":31648,"threads_started":2,"wal-append.queue_time_us":6446}
I20240501 20:11:25.798496 5833 raft_consensus.cc:348] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:25.799134 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(30d2aada073b4d8989d7db8be4034756): 4581 bytes on disk
I20240501 20:11:25.799695 5833 raft_consensus.cc:729] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61380023001a4ee9ada9ec2c6c1b7159, State: Initialized, Role: FOLLOWER
I20240501 20:11:25.800325 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(30d2aada073b4d8989d7db8be4034756) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240501 20:11:25.800885 5833 consensus_queue.cc:260] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6dcaa8efff642dcb7c2c6e8be3a65f7" member_type: VOTER last_known_addr { host: "127.13.251.134" port: 42431 } } peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:25.803195 5833 ts_tablet_manager.cc:1398] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 20:11:25.807250 24130 tablet_copy_service.cc:342] P 8559229e044b4e1ea661b881b3f9af50: Request end of tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-b7afaa4052754794b8f220501887af08 received from {username='slave'} at 127.13.251.132:52271
I20240501 20:11:25.807442 24130 tablet_copy_service.cc:433] P 8559229e044b4e1ea661b881b3f9af50: ending tablet copy session 61380023001a4ee9ada9ec2c6c1b7159-b7afaa4052754794b8f220501887af08 on tablet b7afaa4052754794b8f220501887af08 with peer 61380023001a4ee9ada9ec2c6c1b7159
I20240501 20:11:25.827270 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(748b4fd500f84a359bd8ee05b861300d) complete. Timing: real 0.090s user 0.053s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11448,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":7292,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":863,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":31751,"threads_started":2,"wal-append.queue_time_us":20216}
I20240501 20:11:25.830019 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(748b4fd500f84a359bd8ee05b861300d): 4834 bytes on disk
I20240501 20:11:25.830719 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(748b4fd500f84a359bd8ee05b861300d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240501 20:11:25.834163 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(4d14c3c71012405e81dabba5f127433d): perf score=0.033404
I20240501 20:11:25.866499 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(4d14c3c71012405e81dabba5f127433d) complete. Timing: real 0.032s user 0.029s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":257,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1465,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":967,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":272,"threads_started":1,"wal-append.queue_time_us":364}
I20240501 20:11:25.869081 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(4d14c3c71012405e81dabba5f127433d): 4921 bytes on disk
I20240501 20:11:25.869621 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(4d14c3c71012405e81dabba5f127433d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240501 20:11:25.901821 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:25.909107 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.028043 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.035715 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.085789 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:26.131788 21357 raft_consensus.cc:1203] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 20:11:26.160248 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.183987 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.195211 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.253623 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:26.279995 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:26.282979 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:26.295414 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:26.321566 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:26.345111 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:11:26.349714 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:26.352816 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:26.353663 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:26.363684 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:26.369164 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:26.407359 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:26.407488 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.414788 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.470350 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.495198 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.509826 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:26.599469 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:26.734232 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:26.794008 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:26.828931 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:26.849228 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:26.868476 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:26.884758 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:26.908501 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:26.909701 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:26.925159 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:26.927112 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:26.940544 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:26.977957 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:27.006569 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:27.012076 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.034790 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.036798 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.045403 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.074019 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.095036 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.111954 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.119475 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.123102 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.162151 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.168699 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.174897 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.184324 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.211254 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.243191 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.249377 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.286724 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240501 20:11:27.295089 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240501 20:11:27.308933 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240501 20:11:27.310639 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.311661 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 20:11:27.339660 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240501 20:11:27.341135 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.362210 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.380797 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.403183 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.405957 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:11:27.407461 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:27.417793 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:27.424845 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:27.436611 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:27.446393 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:27.449424 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:27.486645 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:27.489941 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:27.498502 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:27.519781 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:27.568882 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:27.628127 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:27.725854 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:11:27.986451 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:28.030403 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:28.034731 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:28.039330 21034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 52881us
I20240501 20:11:28.055331 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:28.097234 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:28.120114 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:28.134335 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:28.198050 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:28.323325 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:28.357878 24109 consensus_queue.cc:237] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } }
I20240501 20:11:28.360039 23975 raft_consensus.cc:1261] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:28.360278 21357 raft_consensus.cc:1261] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Refusing update from remote peer 8559229e044b4e1ea661b881b3f9af50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:28.360631 5520 consensus_queue.cc:1035] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:28.360914 5989 consensus_queue.cc:1035] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:28.364260 5989 raft_consensus.cc:2932] T b7afaa4052754794b8f220501887af08 P 8559229e044b4e1ea661b881b3f9af50 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:28.364954 21357 raft_consensus.cc:2932] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:28.368836 23975 raft_consensus.cc:2932] T b7afaa4052754794b8f220501887af08 P 744c5a532e724391830c0785d32eb2fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
I20240501 20:11:28.373625 20825 catalog_manager.cc:5484] T b7afaa4052754794b8f220501887af08 P 61380023001a4ee9ada9ec2c6c1b7159 reported cstate change: config changed from index 2 to 3, VOTER c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134) evicted. New cstate: current_term: 1 leader_uuid: "8559229e044b4e1ea661b881b3f9af50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "744c5a532e724391830c0785d32eb2fb" member_type: VOTER last_known_addr { host: "127.13.251.135" port: 41019 } } peers { permanent_uuid: "8559229e044b4e1ea661b881b3f9af50" member_type: VOTER last_known_addr { host: "127.13.251.136" port: 44267 } } peers { permanent_uuid: "61380023001a4ee9ada9ec2c6c1b7159" member_type: VOTER last_known_addr { host: "127.13.251.132" port: 46399 } } }
W20240501 20:11:28.387904 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:28.412750 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet b7afaa4052754794b8f220501887af08 with delete_type TABLET_DATA_TOMBSTONED (TS c6dcaa8efff642dcb7c2c6e8be3a65f7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57076
I20240501 20:11:28.423970 6148 tablet_replica.cc:330] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7: stopping tablet replica
I20240501 20:11:28.424206 6148 raft_consensus.cc:2229] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:28.424412 6148 raft_consensus.cc:2258] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:28.425194 6148 ts_tablet_manager.cc:1875] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:28.431715 6148 ts_tablet_manager.cc:1888] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:11:28.431864 6148 log.cc:1196] T b7afaa4052754794b8f220501887af08 P c6dcaa8efff642dcb7c2c6e8be3a65f7: Deleting WAL directory at /tmp/dist-test-task6p4U5T/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714594080532152-14318-0/raft_consensus-itest-cluster/ts-5/wal/wals/b7afaa4052754794b8f220501887af08
I20240501 20:11:28.432660 20813 catalog_manager.cc:4830] TS c6dcaa8efff642dcb7c2c6e8be3a65f7 (127.13.251.134:42431): tablet b7afaa4052754794b8f220501887af08 (table rebalancer_extra_table_14 [id=835c9ba291964898ba1ee54dcb9b6df1]) successfully deleted
W20240501 20:11:28.438902 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:28.488902 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:28.521294 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:28.557773 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:28.581619 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:28.673496 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:28.724640 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:28.733779 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:28.739485 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:28.797852 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:28.802714 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:28.819880 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:28.831508 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:28.832806 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:28.847280 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:28.857288 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:28.890625 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:28.892462 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:28.896324 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:28.940544 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:28.977207 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.050251 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:29.145933 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:29.159837 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(ca03186ff8b342dd958b205125b14177): perf score=0.033339
W20240501 20:11:29.192612 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.201560 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.229571 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:11:29.251075 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(ca03186ff8b342dd958b205125b14177) complete. Timing: real 0.091s user 0.048s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":349,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":2070,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":938,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":5036,"threads_started":2,"wal-append.queue_time_us":4695}
I20240501 20:11:29.254794 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(ca03186ff8b342dd958b205125b14177): 4834 bytes on disk
I20240501 20:11:29.255789 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(ca03186ff8b342dd958b205125b14177) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240501 20:11:29.290266 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(e1476e8f4c9343928f68365fd2772f0f): perf score=0.033375
W20240501 20:11:29.311259 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:29.336508 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(ca03186ff8b342dd958b205125b14177): perf score=0.033385
I20240501 20:11:29.423002 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(e1476e8f4c9343928f68365fd2772f0f) complete. Timing: real 0.132s user 0.036s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":19856,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":26293,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1214,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29612,"threads_started":2,"wal-append.queue_time_us":15310}
I20240501 20:11:29.430601 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(3a7e956aedc44f56b3bf629c80ff65e4): perf score=0.033411
W20240501 20:11:29.436317 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:29.437645 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:29.479261 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:29.483894 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:29.491974 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:29.506285 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:29.516199 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:29.541877 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:29.542834 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.542956 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.543048 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:29.543238 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:29.562835 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.573220 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.582687 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:29.612833 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(ca03186ff8b342dd958b205125b14177) complete. Timing: real 0.275s user 0.075s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":22238,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1939,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":948,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":35082,"threads_started":2,"wal-append.queue_time_us":11723}
I20240501 20:11:29.615183 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(ca03186ff8b342dd958b205125b14177): 4834 bytes on disk
I20240501 20:11:29.616010 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(ca03186ff8b342dd958b205125b14177) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":176,"lbm_reads_lt_1ms":4}
I20240501 20:11:29.623158 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:29.681109 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.696509 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:29.713292 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(3a7e956aedc44f56b3bf629c80ff65e4) complete. Timing: real 0.282s user 0.046s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":29092,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1665,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":958,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":234,"threads_started":1,"wal-append.queue_time_us":328}
I20240501 20:11:29.715379 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(e1476e8f4c9343928f68365fd2772f0f): 4919 bytes on disk
I20240501 20:11:29.716107 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(e1476e8f4c9343928f68365fd2772f0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240501 20:11:29.718209 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(3a7e956aedc44f56b3bf629c80ff65e4): 4581 bytes on disk
I20240501 20:11:29.719092 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(3a7e956aedc44f56b3bf629c80ff65e4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240501 20:11:29.722673 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.730849 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.743405 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.767504 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.789976 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.793220 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.794521 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.840494 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.852403 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.877389 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.878844 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.910075 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.935745 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.946401 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.951453 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:29.978663 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:30.033495 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:30.096300 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:30.108440 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:30.117228 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:30.141613 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:30.194634 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:30.234823 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:30.255760 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:30.286700 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:30.350615 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:11:30.641623 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:30.715463 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:30.768546 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:30.851233 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:30.862609 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:30.867813 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:30.868333 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:30.879930 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:30.889539 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:30.891652 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:30.936079 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:30.959250 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:31.015764 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:31.082062 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:31.135020 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:31.176676 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:31.177630 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:31.277740 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:31.307163 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:31.335863 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:31.395732 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:31.396358 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:31.405854 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:31.416291 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:31.430553 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:31.473896 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:31.487821 23779 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 129922us
W20240501 20:11:31.502178 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:31.508371 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:31.528227 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:31.566174 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:31.582190 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:31.583068 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:31.624243 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:31.625001 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:31.711266 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:11:31.767927 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:31.846380 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:31.864455 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:31.894383 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:31.946301 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:31.955214 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:31.998070 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:32.006726 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:32.011302 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:32.014320 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:32.033608 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.052990 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:32.070698 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:32.078351 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:32.118330 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:32.118778 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:32.124943 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.165620 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.184612 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.198362 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.199008 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.255203 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.285159 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.301400 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.311347 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:11:32.313366 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.315495 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:11:32.316439 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.323333 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.347592 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 20:11:32.348534 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.363265 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:11:32.364306 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.372946 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:11:32.386090 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:32.396905 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(75a1b78bb9574b588e6bf69ae7e68bc3): perf score=0.033343
W20240501 20:11:32.440780 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.470378 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:32.508522 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(8d75b25758954877ab36e8029fb29463): perf score=0.033375
W20240501 20:11:32.513088 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:32.539763 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(75a1b78bb9574b588e6bf69ae7e68bc3) complete. Timing: real 0.143s user 0.053s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":844,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":2329,"drs_written":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5155,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":9052,"threads_started":2,"wal-append.queue_time_us":8158}
I20240501 20:11:32.542882 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(75a1b78bb9574b588e6bf69ae7e68bc3): 4921 bytes on disk
I20240501 20:11:32.553105 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling FlushMRSOp(7a2f74307a194dc49d184d4ea288b36d): perf score=0.033378
I20240501 20:11:32.554260 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(75a1b78bb9574b588e6bf69ae7e68bc3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":158,"lbm_reads_lt_1ms":4}
I20240501 20:11:32.562709 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling FlushMRSOp(7a2f74307a194dc49d184d4ea288b36d): perf score=0.033382
W20240501 20:11:32.586676 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:11:32.590085 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.658344 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:32.703711 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:32.703894 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:32.714267 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: FlushMRSOp(7a2f74307a194dc49d184d4ea288b36d) complete. Timing: real 0.150s user 0.059s sys 0.037s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":107,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":5348,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1229,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":54572,"threads_started":1,"wal-append.queue_time_us":54229}
I20240501 20:11:32.717036 21013 maintenance_manager.cc:419] P cda19d6a81c0473db302535ce2ada7f0: Scheduling UndoDeltaBlockGCOp(7a2f74307a194dc49d184d4ea288b36d): 4581 bytes on disk
I20240501 20:11:32.720914 20904 maintenance_manager.cc:643] P cda19d6a81c0473db302535ce2ada7f0: UndoDeltaBlockGCOp(7a2f74307a194dc49d184d4ea288b36d) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
W20240501 20:11:32.727607 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:32.736964 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(8d75b25758954877ab36e8029fb29463) complete. Timing: real 0.228s user 0.092s sys 0.036s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66036,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":3361,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":984,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":100297,"threads_started":2,"wal-append.queue_time_us":23214}
I20240501 20:11:32.740226 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling FlushMRSOp(75a1b78bb9574b588e6bf69ae7e68bc3): perf score=0.033435
I20240501 20:11:32.789206 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:32.803695 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: FlushMRSOp(75a1b78bb9574b588e6bf69ae7e68bc3) complete. Timing: real 0.063s user 0.043s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1500,"drs_written":1,"lbm_read_time_us":149,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1039,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6740,"threads_started":1,"wal-append.queue_time_us":6632}
I20240501 20:11:32.806879 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(75a1b78bb9574b588e6bf69ae7e68bc3): 4921 bytes on disk
I20240501 20:11:32.807750 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(75a1b78bb9574b588e6bf69ae7e68bc3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240501 20:11:32.810488 21144 maintenance_manager.cc:419] P feac73db0a3b4aafa374ca5dfd322b08: Scheduling UndoDeltaBlockGCOp(8d75b25758954877ab36e8029fb29463): 4835 bytes on disk
I20240501 20:11:32.811322 21035 maintenance_manager.cc:643] P feac73db0a3b4aafa374ca5dfd322b08: UndoDeltaBlockGCOp(8d75b25758954877ab36e8029fb29463) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4}
I20240501 20:11:32.823724 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:32.849313 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:11:32.857000 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: FlushMRSOp(7a2f74307a194dc49d184d4ea288b36d) complete. Timing: real 0.303s user 0.061s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13409,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":7214,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1151,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":68955,"threads_started":2,"wal-append.queue_time_us":45164}
I20240501 20:11:32.862896 21404 maintenance_manager.cc:419] P 61380023001a4ee9ada9ec2c6c1b7159: Scheduling UndoDeltaBlockGCOp(7a2f74307a194dc49d184d4ea288b36d): 4581 bytes on disk
I20240501 20:11:32.865891 21295 maintenance_manager.cc:643] P 61380023001a4ee9ada9ec2c6c1b7159: UndoDeltaBlockGCOp(7a2f74307a194dc49d184d4ea288b36d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240501 20:11:32.866387 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:32.869776 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:32.890702 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:32.903380 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:32.949132 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:33.051927 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:33.339215 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:33.353763 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:33.360076 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:33.378353 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:33.405691 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:33.406895 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:33.417135 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:33.427745 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 146371us
W20240501 20:11:33.457216 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:33.460896 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:33.477114 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:33.486141 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:33.489357 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:33.612548 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:33.612571 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:33.629379 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:33.694666 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:33.709576 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:33.728361 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:33.750490 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:11:33.857007 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:33.870735 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:33.908704 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:33.926388 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:33.934144 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:33.954342 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:33.966943 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:11:33.972065 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:33.981995 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:33.998034 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:34.068897 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:34.085928 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:34.161007 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.253978 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.286933 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:34.308548 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:34.320456 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.347828 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.355161 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.425712 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.430989 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.487242 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:34.493028 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.529157 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.564085 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.579893 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.588804 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.693464 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:11:34.694299 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:34.697196 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:34.704545 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.723318 23911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 189215us
I20240501 20:11:34.741135 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:34.749635 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:34.756960 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:34.758925 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:34.758970 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.764492 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.781898 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:11:34.820344 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:34.856626 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.886317 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.920207 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.928555 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:34.957664 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:35.006028 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:35.010327 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:35.033717 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:35.050596 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:35.066785 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:35.148296 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:11:35.248198 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:35.251158 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:35.266755 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:35.307432 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:35.331632 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:11:35.331787 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:35.366290 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:35.366339 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:35.445032 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:35.460682 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:11:35.480022 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:35.570616 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:35.732260 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:11:35.781122 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:35.793224 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:35.793717 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:35.804215 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:11:35.819695 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:35.835031 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:35.859376 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:35.865391 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:35.894618 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:35.952121 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:36.051293 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.077483 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.103258 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.148305 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.167423 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.213441 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.346835 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.363675 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.372817 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.393695 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.438086 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.442473 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.502864 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.508965 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:36.537847 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:36.550668 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:36.558315 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:36.575788 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:36.584128 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:36.608897 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:36.633783 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:36.700443 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.758706 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.779279 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.805557 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.830989 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.850082 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:36.879449 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.951253 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:36.973973 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:11:37.014165 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:37.014575 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:37.015789 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:37.038659 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.040331 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:37.045231 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:11:37.077334 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:37.091450 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:11:37.095655 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:37.138527 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:11:37.142722 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:37.156615 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.164533 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.164690 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.176694 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:11:37.181280 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:37.232084 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.292507 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.316035 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:11:37.317399 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.327006 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:11:37.348031 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.429347 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:11:37.438282 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240501 20:11:37.487285 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:37.490429 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:37.490478 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:37.508942 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:37.532939 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:37.537505 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:37.537892 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 20:11:37.561556 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:37.565029 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:37.575208 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.579263 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.586064 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.613505 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.617127 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.624701 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240501 20:11:37.627925 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.661923 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.675662 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.881986 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:37.923641 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:38.037539 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:38.121791 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:38.164286 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:11:38.258666 23910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 63661.8us
W20240501 20:11:38.259032 23647 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 59610.9us
I20240501 20:11:38.306464 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:38.313601 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:38.330533 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:38.337086 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:38.387039 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:11:38.395336 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:38.395941 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:38.434307 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:38.435843 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:38.456988 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:38.495718 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:38.528112 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:38.589563 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:38.616935 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:38.640204 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:38.660116 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:38.786621 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:38.797117 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:11:38.801506 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:38.815246 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:38.827782 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:38.864068 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:38.866510 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:38.871978 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:38.901050 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:38.901245 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:38.903730 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:11:38.910828 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:38.911438 24044 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 62703.6us
W20240501 20:11:38.916078 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:39.004047 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.018319 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.041492 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.057790 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.096832 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:39.127750 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:39.170737 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.268841 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:39.308614 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:39.312637 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:39.315834 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:39.321902 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:39.342226 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:39.377543 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:39.401089 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:39.411303 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:39.459264 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.512315 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:39.538295 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.560570 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.560910 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.568786 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.618800 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.621170 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.667459 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.731245 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:39.749176 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.757994 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.765714 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.790807 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.823892 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.830796 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:39.903246 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:39.983386 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:40.035867 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:40.046043 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:40.065832 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:40.084849 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:40.126634 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:40.132876 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:40.143954 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:40.148936 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 48170.2us
W20240501 20:11:40.158438 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:40.176781 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:40.215178 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:40.235108 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:40.252548 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:40.273059 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:40.415273 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:40.424901 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:40.436986 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling FlushMRSOp(aebb2a23076449fd9db492ed3e3b0cbf): perf score=0.033398
I20240501 20:11:40.564337 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: FlushMRSOp(aebb2a23076449fd9db492ed3e3b0cbf) complete. Timing: real 0.127s user 0.086s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":339,"dirs.queue_time_us":15378,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":2234,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1470,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17381,"threads_started":3,"wal-append.queue_time_us":6373}
I20240501 20:11:40.578437 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling UndoDeltaBlockGCOp(aebb2a23076449fd9db492ed3e3b0cbf): 4581 bytes on disk
I20240501 20:11:40.579465 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: UndoDeltaBlockGCOp(aebb2a23076449fd9db492ed3e3b0cbf) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":169,"lbm_reads_lt_1ms":4}
W20240501 20:11:40.730157 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:40.731736 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:40.770789 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:40.807561 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:40.834800 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:11:40.846230 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:40.903622 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:40.905622 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:40.938694 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:40.950716 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:40.951783 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:40.952175 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:41.008092 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:41.042201 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:41.067721 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.131004 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:41.220511 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.285451 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:41.308424 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling FlushMRSOp(30d2aada073b4d8989d7db8be4034756): perf score=0.033347
W20240501 20:11:41.349241 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:41.350878 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:41.385922 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:41.461962 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: FlushMRSOp(30d2aada073b4d8989d7db8be4034756) complete. Timing: real 0.153s user 0.063s sys 0.025s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":350,"dirs.queue_time_us":8191,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1110,"drs_written":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1609,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":15268,"threads_started":3,"wal-append.queue_time_us":6521}
I20240501 20:11:41.466681 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling UndoDeltaBlockGCOp(30d2aada073b4d8989d7db8be4034756): 4581 bytes on disk
W20240501 20:11:41.466876 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:11:41.469673 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling FlushMRSOp(c1e295a46a174f3f8673e03f57890b58): perf score=0.033390
I20240501 20:11:41.475171 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: UndoDeltaBlockGCOp(30d2aada073b4d8989d7db8be4034756) complete. Timing: real 0.008s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240501 20:11:41.478592 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling FlushMRSOp(a115fe66d11143ea8e24aa891e4e2f33): perf score=0.033367
W20240501 20:11:41.524482 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:41.534901 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.597736 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:41.607384 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.616119 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:41.632308 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:41.687997 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: FlushMRSOp(a115fe66d11143ea8e24aa891e4e2f33) complete. Timing: real 0.209s user 0.043s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1726,"drs_written":1,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6337,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10042,"threads_started":1,"wal-append.queue_time_us":6428}
W20240501 20:11:41.688719 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:11:41.690599 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling UndoDeltaBlockGCOp(a115fe66d11143ea8e24aa891e4e2f33): 4921 bytes on disk
I20240501 20:11:41.692116 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: UndoDeltaBlockGCOp(a115fe66d11143ea8e24aa891e4e2f33) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240501 20:11:41.727466 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.735086 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.833432 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.881728 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.904614 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.913748 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.950811 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:41.953662 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:42.020532 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.087891 24045 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 125670us
W20240501 20:11:42.089362 24043 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 111777us
W20240501 20:11:42.114847 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.129664 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 48859.2us
W20240501 20:11:42.141187 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.141429 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:11:42.156759 24047 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.447s user 0.098s sys 0.146s
I20240501 20:11:42.162415 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: FlushMRSOp(c1e295a46a174f3f8673e03f57890b58) complete. Timing: real 0.691s user 0.171s sys 0.168s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":18774,"dirs.queue_time_us":3692,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":3002,"drs_written":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"lbm_write_time_us":918,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":469765,"threads_started":3,"wal-append.queue_time_us":438994}
I20240501 20:11:42.166060 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling UndoDeltaBlockGCOp(c1e295a46a174f3f8673e03f57890b58): 4834 bytes on disk
I20240501 20:11:42.168701 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: UndoDeltaBlockGCOp(c1e295a46a174f3f8673e03f57890b58) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 20:11:42.171139 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling FlushMRSOp(43e80df9af0e40c5acc4dea95d7e95a8): perf score=0.033557
W20240501 20:11:42.171384 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.171838 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.172472 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.195459 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 65510.9us
W20240501 20:11:42.239912 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.245746 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.254065 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:11:42.332274 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: FlushMRSOp(43e80df9af0e40c5acc4dea95d7e95a8) complete. Timing: real 0.160s user 0.044s sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":16638,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1711,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":934,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":56992,"threads_started":2,"wal-append.queue_time_us":40664}
I20240501 20:11:42.334810 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling UndoDeltaBlockGCOp(43e80df9af0e40c5acc4dea95d7e95a8): 4581 bytes on disk
I20240501 20:11:42.335631 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: UndoDeltaBlockGCOp(43e80df9af0e40c5acc4dea95d7e95a8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
W20240501 20:11:42.340276 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:11:42.341164 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.341871 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.343796 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:11:42.344676 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:11:42.385834 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:42.401333 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:42.407524 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:42.414844 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:42.423194 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:42.441371 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240501 20:11:42.455163 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:42.461782 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
I20240501 20:11:42.468560 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:42.486308 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:42.562726 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240501 20:11:42.584262 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.590675 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.657234 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.709225 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:11:42.760592 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.785539 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.864718 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.882946 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:42.978083 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:43.013106 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:43.114336 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling FlushMRSOp(7a2f74307a194dc49d184d4ea288b36d): perf score=0.033339
W20240501 20:11:43.139400 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:43.144003 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:11:43.207795 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling FlushMRSOp(4587a9d5392742b58ec8d64619f2dda1): perf score=0.033350
I20240501 20:11:43.231743 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:43.243532 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:43.247900 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:43.265386 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: FlushMRSOp(7a2f74307a194dc49d184d4ea288b36d) complete. Timing: real 0.151s user 0.059s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18067,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":13079,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":22066,"threads_started":2,"wal-append.queue_time_us":4258}
I20240501 20:11:43.268002 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling UndoDeltaBlockGCOp(7a2f74307a194dc49d184d4ea288b36d): 4581 bytes on disk
I20240501 20:11:43.268648 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: UndoDeltaBlockGCOp(7a2f74307a194dc49d184d4ea288b36d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 20:11:43.281778 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:43.311827 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:11:43.328428 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: FlushMRSOp(4587a9d5392742b58ec8d64619f2dda1) complete. Timing: real 0.120s user 0.040s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":377,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":15106,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":863,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28908,"threads_started":2,"wal-append.queue_time_us":28197}
I20240501 20:11:43.334062 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling UndoDeltaBlockGCOp(4587a9d5392742b58ec8d64619f2dda1): 4581 bytes on disk
I20240501 20:11:43.340261 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: UndoDeltaBlockGCOp(4587a9d5392742b58ec8d64619f2dda1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
W20240501 20:11:43.354734 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:11:43.376057 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:43.400883 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:43.403400 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:43.413889 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:43.466665 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:43.488337 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:43.611833 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:43.716226 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:43.735419 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:43.750337 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:11:43.816746 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:43.846406 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:43.849545 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:43.850596 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:43.872937 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:43.873293 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:43.889946 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:43.902369 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:43.922576 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:43.922709 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:43.942811 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:43.962702 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:44.027855 21031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 128886us
W20240501 20:11:44.030189 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.049796 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:44.078555 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:44.085026 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:11:44.147588 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling FlushMRSOp(420b46edf5fc438fb01e822440bce8f1): perf score=0.033336
W20240501 20:11:44.159022 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:11:44.218730 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: FlushMRSOp(420b46edf5fc438fb01e822440bce8f1) complete. Timing: real 0.071s user 0.053s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":380,"dirs.queue_time_us":6360,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":6619,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":854,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11614,"threads_started":3,"wal-append.queue_time_us":4863}
I20240501 20:11:44.222445 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling UndoDeltaBlockGCOp(420b46edf5fc438fb01e822440bce8f1): 4921 bytes on disk
I20240501 20:11:44.223389 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: UndoDeltaBlockGCOp(420b46edf5fc438fb01e822440bce8f1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
W20240501 20:11:44.290232 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.302920 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:44.308702 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.352945 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:44.384331 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.408881 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.430073 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.446062 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.466428 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.516454 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.593086 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.613691 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.620076 24043 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 50349.2us
W20240501 20:11:44.624071 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.624320 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.648696 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.671286 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:44.713725 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:44.716728 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:44.722313 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:44.735204 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:44.752701 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:44.757465 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:44.767145 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.805007 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:44.805768 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:44.824599 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:44.865589 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:44.951078 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:44.953325 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.956246 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.960758 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:44.975400 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:45.171270 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:45.203373 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:45.339780 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:45.381212 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:45.418277 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:45.420426 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:45.432438 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:45.449278 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:45.468586 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:45.471769 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:45.490113 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:45.547152 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:45.564587 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:45.579114 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:45.606823 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:45.620922 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:45.778614 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:45.846805 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:45.857303 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:45.923082 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:45.931265 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:45.931968 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:45.940186 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:45.946903 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:45.979480 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:45.979944 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:46.026423 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:46.053465 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:46.056948 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:46.171126 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:46.223373 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:46.304349 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:46.403707 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:46.405535 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:46.426436 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:11:46.454284 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:46.465320 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:46.537087 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:46.545276 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:46.555979 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:46.588402 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:46.608997 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:46.615497 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:46.616899 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:46.622589 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:46.639675 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:46.640551 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:46.654685 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:46.684511 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:11:46.693092 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:46.718335 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:46.738274 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:46.808789 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:46.810573 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:46.875576 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.028441 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.028568 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:47.029130 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.083941 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.098688 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.100811 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.127002 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.169492 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.218309 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.226634 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.230681 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.256960 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.272953 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:11:47.285138 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:47.308781 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:11:47.312903 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:47.321050 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:47.335969 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.338598 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:11:47.344187 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:47.362103 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:11:47.363662 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:11:47.366480 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:47.375144 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:47.401074 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:47.417313 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:11:47.459131 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:11:47.473246 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:11:47.540685 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.546074 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:47.572455 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 20:11:47.730352 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.741976 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:11:47.746199 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:11:47.797082 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:47.820729 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:47.840241 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:47.859090 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:47.860630 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:47.899276 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:11:47.935837 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:47.947257 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:47.966097 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:11:47.969472 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:47.977849 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.013789 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.129614 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:48.194483 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.266350 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.280210 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.341625 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:11:48.351074 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:48.380017 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:48.384436 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:48.417178 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:48.496292 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:48.518741 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:48.528118 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:48.536041 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:48.539047 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:48.539284 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.578780 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.639319 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:48.697309 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.704820 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:48.862718 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:48.909377 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.910535 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:48.932673 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:49.070233 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:11:49.174747 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:49.175824 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:49.190301 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.203825 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:49.232694 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:49.235229 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:49.244076 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:49.255817 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:49.302498 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:49.309635 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:11:49.314204 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:49.332399 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:49.343627 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:49.378572 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.385380 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:49.400120 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.417379 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.419577 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.433794 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.595427 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.596590 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.608506 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.643525 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:49.658958 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.675176 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.677187 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.697671 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.728312 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.740399 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.754125 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.759850 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:49.802235 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:49.830248 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:49.840811 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:49.841475 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:49.854008 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:49.862886 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:49.882134 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:49.912856 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:49.913672 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:49.918505 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:49.946863 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:50.189561 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:50.201447 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:50.207041 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:50.280364 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:50.329900 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:50.347126 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:50.381933 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:50.384826 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:50.399749 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:50.418774 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:50.456645 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:50.471277 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:50.484207 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:50.526563 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:50.620851 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:50.733410 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:50.844779 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:50.934954 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:50.982506 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:50.982738 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:50.998438 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:51.084218 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:51.097755 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:51.131505 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:51.159214 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:51.172246 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:51.174034 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:51.206983 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:51.213335 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:51.242761 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:51.250125 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:51.262203 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:51.278991 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:51.284708 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:51.320389 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:51.507352 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:51.549311 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:51.574664 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:51.574800 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:51.593567 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:51.596024 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:51.722800 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:51.749735 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:51.757058 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:51.772145 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:51.778383 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:51.780575 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:11:51.807556 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:51.815934 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:51.852909 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:51.860754 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:11:51.870402 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:51.921023 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:51.942631 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:51.958494 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:51.961495 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:51.999657 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:52.009382 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.039475 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.047999 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.077337 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.099978 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.103921 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.122630 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.142649 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.145630 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.146924 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.269026 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.274897 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:11:52.275823 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:52.303105 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 51147.3us
W20240501 20:11:52.317201 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:11:52.317564 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:52.331130 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:52.341873 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:52.363431 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:52.387032 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:11:52.387975 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.419929 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:11:52.435817 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:52.436933 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:52.454501 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.469936 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.477905 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 20:11:52.491077 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:11:52.500399 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.502806 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.524734 21034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 46833.3us
W20240501 20:11:52.564314 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:52.567660 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.592360 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:11:52.662873 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 20:11:52.750221 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:11:52.768918 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.815980 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.950759 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:52.983255 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:52.989022 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:11:53.120445 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:53.170001 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:53.198453 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:53.214349 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:53.215291 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:11:53.215616 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:53.249933 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:53.265580 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:11:53.315783 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:53.346653 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:53.359099 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:53.408952 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:53.412443 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:53.470069 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:53.557816 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:11:53.652882 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:53.677284 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:53.692150 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:53.698597 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:53.714191 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:53.715091 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:53.739795 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:53.772083 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:53.801961 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:53.810813 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:53.866902 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:53.879967 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:53.903761 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:53.907581 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:53.932472 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:53.978062 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:54.011374 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:11:54.216084 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:54.281592 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:54.290060 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:54.305326 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:11:54.341763 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:54.351079 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:54.373032 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:11:54.406811 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:54.406968 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:54.411238 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:54.414461 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:54.424093 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.432412 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.437743 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.469949 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.474509 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:11:54.513988 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.553797 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.575935 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:54.640689 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.647379 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.652398 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.663379 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.663746 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.724820 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.757490 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.761827 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.777168 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.802497 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.854873 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.897032 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.902117 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:54.939101 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:54.967052 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:54.997422 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:54.998310 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:55.060190 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:55.062283 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:11:55.064298 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:55.080204 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.122700 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 101385us
I20240501 20:11:55.122828 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:55.128023 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.144946 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:11:55.194164 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:55.229552 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.316210 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.397246 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.433410 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:55.512987 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:11:55.563584 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:55.569470 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:55.581388 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:55.606971 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:55.630264 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:55.643934 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:55.731452 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:55.747501 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.755751 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.826298 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.862488 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.893262 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.924925 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:55.958750 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:56.006667 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:11:56.039112 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:56.057685 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:56.088028 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:56.129664 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:11:56.135917 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:56.158185 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:56.188793 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:56.217821 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:56.301558 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:56.305188 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.398445 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.417408 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.458221 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:56.490069 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.520308 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:56.565845 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:56.609673 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:11:56.659696 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.683820 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:56.718890 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.770836 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:11:56.788209 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:56.796928 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:56.833642 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:56.837054 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:56.852646 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:11:56.876087 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:56.880978 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:56.908318 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.908960 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.910015 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:11:56.923053 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:56.951290 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.955996 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:56.979805 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.133180 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.138518 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.178172 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:57.206051 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.214365 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.219452 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.232831 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:57.236604 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.242322 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.260715 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.264294 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:11:57.309899 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:57.309952 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:57.314517 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:57.322073 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:57.343714 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:57.376515 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:11:57.378211 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:57.381199 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:57.405686 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.438625 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.493330 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 20:11:57.496575 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.505780 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.549554 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 20:11:57.550700 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:57.553965 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:11:57.676229 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 20:11:57.677120 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.681262 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:11:57.786038 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:11:57.791199 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:57.807960 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:57.813802 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:11:57.815788 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:11:57.820079 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:57.827193 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:11:57.833971 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:57.840433 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:11:57.851703 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:57.877965 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:57.915169 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:57.918602 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.009676 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:58.186902 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.194522 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.219632 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.246215 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.344774 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.408427 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:11:58.466063 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:58.489840 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:11:58.501046 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:58.531379 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:58.533838 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:58.538532 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:11:58.587445 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:11:58.596870 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:58.611410 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.714574 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.811486 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.849864 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.884023 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:58.929059 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:58.931396 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:11:58.942495 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:58.966387 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:11:58.971760 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:11:58.981088 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:58.987849 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:59.021351 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:59.029983 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.046105 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.046268 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:11:59.046357 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:11:59.056201 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:11:59.067183 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:59.120313 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:11:59.151091 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:59.282549 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.333667 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.388576 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.404606 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.443746 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.445641 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:11:59.465687 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:11:59.474867 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:11:59.478936 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:11:59.485402 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.485976 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:11:59.519843 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:59.522110 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:59.524372 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:11:59.528374 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:11:59.528836 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:11:59.568110 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.623142 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.645330 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:11:59.652624 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.657109 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.694996 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.797331 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.812922 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.833595 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.885536 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.913166 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.942787 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:11:59.960757 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.025851 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.050194 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:00.061758 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.099891 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.101774 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.155979 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.197899 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:12:00.247491 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:00.252406 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:00.254096 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:00.267339 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:00.273562 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:00.309754 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:00.316759 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:00.462558 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.504851 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.533367 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.638108 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:00.642160 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.661810 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.749047 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:00.749603 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:12:00.750757 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:00.777107 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:00.791489 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:00.820787 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:00.835381 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:00.835912 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:00.855800 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:12:00.888649 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:00.942440 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:01.129770 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:01.142536 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:12:01.258662 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:01.286900 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:12:01.311400 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:01.325183 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:01.339294 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:01.359846 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:01.376547 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:01.392444 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:01.396996 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:01.415277 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:12:01.419466 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:01.476877 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:01.482976 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:12:01.485244 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:01.516124 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:01.537165 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:01.597846 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:01.619387 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:01.657035 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:01.669091 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:01.723124 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:01.818222 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:01.860734 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:01.893766 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:12:01.975845 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:01.979022 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:01.982673 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:01.999866 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.021908 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:02.023710 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:02.044543 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:02.059432 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.067333 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:02.071801 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:02.100953 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:02.106294 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:02.132409 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.172683 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.193352 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:12:02.205835 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.237376 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:02.306007 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.316045 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.359050 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.442060 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:02.513485 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:02.525745 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:02.531924 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:12:02.537362 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:02.540516 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:02.547386 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:02.548413 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:02.556814 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:02.566424 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:02.587074 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
I20240501 20:12:02.589319 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:02.595687 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:12:02.602672 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.602803 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.675594 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.687034 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.687949 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.712332 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 20:12:02.713315 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.824316 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 20:12:02.851243 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:02.913071 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.922173 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:12:02.923120 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:02.971959 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:03.001809 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:03.002032 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 45051.1us
I20240501 20:12:03.060192 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:03.060408 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:03.092168 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:03.147197 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:03.169950 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:03.188869 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:03.201049 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:03.239689 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:03.248490 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:03.258946 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:03.272328 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:03.384418 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:03.386673 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:03.597119 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:03.788424 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:03.819414 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:12:03.820120 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:03.826658 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:03.843724 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:03.869554 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:03.875963 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:03.879483 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:03.893565 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:03.912092 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:03.913707 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:03.923305 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:03.932487 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:03.950577 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:03.958396 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:03.961311 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:03.970954 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.034621 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.047645 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:04.097658 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:04.143200 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.152153 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:04.157671 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.300163 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:04.359651 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:04.361027 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:04.370524 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:04.371273 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:04.395867 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:04.396162 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:04.420163 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:04.436689 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:04.466704 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:04.527783 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.548669 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.558269 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.561152 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.589799 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.594368 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.699023 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.716374 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.742705 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.751112 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:04.790555 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:04.795395 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:04.796809 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:04.814399 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:04.819839 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:04.821517 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:04.850267 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:04.926872 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:04.951665 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:12:04.953290 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.040608 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.044373 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:05.047133 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.075096 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.219695 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.230257 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.236956 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.263974 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.308362 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.399703 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.418439 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.449080 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.464226 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.484792 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:05.518374 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:05.524376 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:05.525327 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:05.562355 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:05.568197 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:05.612654 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:05.632921 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:05.637241 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:05.660387 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:05.666132 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:05.695835 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 76358.2us
W20240501 20:12:05.791206 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.818037 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.923683 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:05.923856 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:05.988149 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling FlushMRSOp(4d14c3c71012405e81dabba5f127433d): perf score=0.033391
I20240501 20:12:06.016517 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:06.060636 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling FlushMRSOp(f392e77df7ff4da489b4d8a8dc415af0): perf score=0.033381
I20240501 20:12:06.086462 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:06.125399 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:06.132550 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:06.139568 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:06.147207 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.181109 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:12:06.182617 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:06.214241 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:06.232017 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:06.292855 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: FlushMRSOp(f392e77df7ff4da489b4d8a8dc415af0) complete. Timing: real 0.232s user 0.066s sys 0.068s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":84547,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":4798,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2311,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":91455,"threads_started":2,"wal-append.queue_time_us":6872}
I20240501 20:12:06.296309 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling UndoDeltaBlockGCOp(f392e77df7ff4da489b4d8a8dc415af0): 4834 bytes on disk
I20240501 20:12:06.308156 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: UndoDeltaBlockGCOp(f392e77df7ff4da489b4d8a8dc415af0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240501 20:12:06.315837 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:12:06.343137 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: FlushMRSOp(4d14c3c71012405e81dabba5f127433d) complete. Timing: real 0.355s user 0.039s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12150,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":2153,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8502,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":23919,"threads_started":2,"wal-append.queue_time_us":1321}
I20240501 20:12:06.349159 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling UndoDeltaBlockGCOp(4d14c3c71012405e81dabba5f127433d): 4921 bytes on disk
I20240501 20:12:06.361063 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: UndoDeltaBlockGCOp(4d14c3c71012405e81dabba5f127433d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240501 20:12:06.381897 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:12:06.382293 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:06.403146 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.449795 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:06.474388 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:06.476724 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.540529 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:06.560760 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.571202 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.576769 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.665234 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:06.772939 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:06.782085 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:06.790880 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:06.805980 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:06.810599 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:06.817811 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:12:06.825742 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:06.877291 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:06.905493 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.920504 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.967754 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.982574 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.996208 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:06.998088 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:07.071275 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.075901 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.095818 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.098399 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.179313 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.196821 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.316891 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:07.355316 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.384078 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:12:07.403117 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:07.415612 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:12:07.424552 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:07.468664 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:07.473429 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:07.475054 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:07.514376 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:07.518755 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:07.518951 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:07.528630 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.535936 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.537928 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.572906 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:07.588068 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:12:07.643332 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.646775 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.660143 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.760515 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:07.786507 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:12:07.788746 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 20:12:07.850857 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240501 20:12:07.917229 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:07.929262 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:07.938745 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:07.946084 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:12:07.946897 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:07.973009 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:07.977614 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:12:07.978461 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:07.993860 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:08.016360 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:12:08.020655 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:08.027609 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.059535 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:08.095332 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.112440 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.165916 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.166929 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.282014 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.405162 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.410550 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:08.477571 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:08.509658 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:08.531049 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:08.548262 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:08.561951 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:08.590837 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:08.591653 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:08.623366 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:08.624862 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:08.662590 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.702549 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.767973 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.795120 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:12:08.803052 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:08.849562 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:08.936793 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:09.029768 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:09.094118 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.139103 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.154072 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.174711 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.190099 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:09.268586 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.293855 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:09.303764 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:09.325516 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:09.328886 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:09.343765 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:09.365365 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:09.377601 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:09.420311 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:12:09.442761 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:09.457828 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:09.476580 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.536303 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.561558 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.655181 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.683882 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.699720 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.700742 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.774454 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.802100 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.833714 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:09.838793 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:09.850870 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:09.854264 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:09.890885 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:09.896533 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.912672 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:09.913236 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:09.931306 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:09.931699 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:09.937371 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:09.983798 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:09.999720 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:10.028149 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:10.107012 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.167322 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.187824 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.292284 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.295326 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.330534 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:10.387187 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:10.391240 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:10.438370 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:10.438735 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:10.450843 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:10.469702 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:10.480432 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.503671 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:10.506392 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:10.525516 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.562796 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:10.564138 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.573276 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.611300 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.672464 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.805073 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:10.811686 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.854009 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.903270 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:10.971928 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:11.083380 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:11.108388 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:11.123833 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:11.195935 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:11.197948 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:11.218626 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:11.246837 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:11.259281 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:11.270421 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:11.293787 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:11.325811 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:11.373540 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:11.393245 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:11.503055 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:11.544350 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:11.552222 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:11.580360 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:11.625159 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:12:11.687584 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:11.732118 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:11.760998 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:11.761585 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:11.800537 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:11.812454 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:11.819483 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:11.822721 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:12:11.827925 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:11.857975 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:11.868530 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:11.882089 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:11.889902 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:11.913395 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling FlushMRSOp(75fa04c8726c4f2e932740cc6fbdcdd0): perf score=0.033388
W20240501 20:12:11.980791 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:11.991891 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: FlushMRSOp(75fa04c8726c4f2e932740cc6fbdcdd0) complete. Timing: real 0.078s user 0.047s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":401,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":11389,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":904,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":18442,"threads_started":2,"wal-append.queue_time_us":18303}
I20240501 20:12:11.994640 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling UndoDeltaBlockGCOp(75fa04c8726c4f2e932740cc6fbdcdd0): 4921 bytes on disk
I20240501 20:12:11.995544 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: UndoDeltaBlockGCOp(75fa04c8726c4f2e932740cc6fbdcdd0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4}
W20240501 20:12:12.001773 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.052166 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.063086 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.072175 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:12.187348 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.193985 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.198345 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.259958 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.264397 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:12.304210 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:12.318549 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:12.322971 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:12.327601 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:12.330415 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:12.330672 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:12.340068 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:12.340394 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:12.344480 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:12.376525 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:12.389003 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.407225 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:12.415861 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:12.502784 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling FlushMRSOp(72726d6b786e4c72ace7359101cec949): perf score=0.033374
W20240501 20:12:12.535163 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.540352 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:12.540781 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: FlushMRSOp(72726d6b786e4c72ace7359101cec949) complete. Timing: real 0.038s user 0.026s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6304,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":5235,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":694,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6674,"threads_started":2,"wal-append.queue_time_us":351}
I20240501 20:12:12.542951 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling UndoDeltaBlockGCOp(72726d6b786e4c72ace7359101cec949): 4834 bytes on disk
I20240501 20:12:12.543586 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: UndoDeltaBlockGCOp(72726d6b786e4c72ace7359101cec949) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240501 20:12:12.581463 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:12.594967 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:12.597043 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.623642 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:12:12.695938 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.777328 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.796401 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.809151 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 20:12:12.810086 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:12.814560 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:12:12.954072 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:12.978663 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:12.997172 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:13.010643 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:13.011432 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:13.025842 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:13.033191 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:13.045886 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
I20240501 20:12:13.050101 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:13.054677 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:13.054816 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:13.063262 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:13.073251 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:12:13.074213 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:13.106995 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:13.117511 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:13.166690 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:13.215072 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:13.242136 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:13.401971 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:13.405771 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:13.421352 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:13.489390 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:13.503041 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:13.503088 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:13.526984 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:13.534267 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:13.543666 21033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 56858.2us
I20240501 20:12:13.549376 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:13.551848 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:13.580406 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:13.705670 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:13.775467 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:13.817653 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:13.948510 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:13.969774 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:13.971684 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:13.985206 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:13.996461 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:14.015142 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:14.060894 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:14.064532 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:14.083243 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:14.103394 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:12:14.113296 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.143035 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:14.146939 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:14.155301 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:14.177341 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:14.219661 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.244347 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:14.307689 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:14.335973 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.521812 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.532303 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:14.566483 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.578022 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.594157 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.616636 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.646101 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.768904 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.798079 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.828730 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:14.832731 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:14.845197 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.852412 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:14.861755 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:14.862855 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:14.870587 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:14.877269 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:14.913575 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:14.930662 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.944751 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.969566 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:12:14.971810 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:14.974874 23910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 59051.6us
I20240501 20:12:14.985857 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:15.064872 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:15.161263 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.166568 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.185582 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.192128 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.193220 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.274833 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:15.318759 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:15.324205 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:15.332156 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:15.357303 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.365444 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:15.387243 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:15.398582 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:15.424710 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:15.442507 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:15.479715 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.497758 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.558442 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.561049 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:15.585889 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.595014 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:15.663497 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:15.823400 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:15.824780 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:15.830552 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:15.836486 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:15.837894 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:15.846568 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:15.862360 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:15.865154 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:15.885097 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:15.981978 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:16.040802 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:16.062815 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:16.085795 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:16.195847 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:16.229364 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:16.245815 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:16.457044 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:16.483814 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:16.484491 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:16.488596 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:16.492518 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:16.502166 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:16.503499 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:16.512658 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:16.517127 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:16.551847 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:16.557746 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:12:16.559875 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:16.616643 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:16.689661 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:16.698792 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:16.717588 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:16.747802 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:16.818497 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:16.850396 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:16.889551 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:12:17.003360 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:17.010444 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:17.034890 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:17.046572 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:17.048288 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:17.054463 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:17.057281 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.116248 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:17.124096 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:17.140877 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:17.173197 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:17.178100 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.196942 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.318490 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.340580 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.350911 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.363251 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.370054 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.447741 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:17.502426 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.528266 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:17.533898 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:17.539527 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:17.541024 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:17.576709 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:17.586176 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:17.600848 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:17.614615 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:17.629820 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:17.635337 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:17.637712 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:12:17.646497 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:17.686635 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:17.700942 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.738493 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.756893 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.792052 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.810194 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 20:12:17.811370 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.860327 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:12:17.868089 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.870595 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:17.921838 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.960446 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.980127 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.984333 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:17.993150 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:18.082693 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 20:12:18.087324 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:12:18.093006 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:18.264608 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:18.270975 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:18.307608 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:18.323976 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:18.334192 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:18.345582 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:18.356799 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:18.372388 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:18.397596 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:18.401849 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:18.402535 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:18.432729 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:18.470140 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:18.625584 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:18.652926 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:18.733363 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:18.786818 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:18.799974 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:18.822530 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:18.831281 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:18.860399 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:18.866479 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:18.870343 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:18.890025 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:18.932375 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:18.951200 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:18.998812 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:18.999604 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.010115 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.080330 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.152932 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:12:19.206872 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:19.219605 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.228386 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:19.290225 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:19.300757 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:19.311913 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:19.324666 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:19.329746 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:19.379781 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:19.403867 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:19.427307 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.487306 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.501181 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.505656 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:19.509838 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:19.544550 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.577277 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:19.773221 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.777159 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.823067 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.883479 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.929751 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:19.937521 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:19.950345 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:19.961328 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:19.980943 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:19.982937 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:19.985265 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.033394 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:20.034708 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:20.038357 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 46579.1us
W20240501 20:12:20.041268 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:20.074129 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:20.088976 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:20.105619 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.117473 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.124766 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:12:20.162036 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:20.188990 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.257207 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.295768 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.351976 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.363101 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.401590 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.412758 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:20.450306 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:20.542685 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:20.543294 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:20.566524 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:20.583659 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:20.588824 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:20.601039 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:20.614966 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:20.632555 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:20.680642 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.702618 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.736943 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.739915 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:20.742755 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.864152 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:20.958931 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:20.961828 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:20.983711 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:20.984261 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:20.986164 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:20.990265 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:21.004815 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:21.009049 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:21.017277 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:21.056334 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:21.056614 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:21.076965 23912 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 52351.3us
W20240501 20:12:21.092113 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:21.146040 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:21.155800 23909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 130676us
W20240501 20:12:21.217026 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:21.233392 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:21.338923 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:21.399034 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:21.445885 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:21.524617 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:21.598721 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:21.622711 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:12:21.701215 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:21.705189 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:21.727021 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:21.781530 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:21.782049 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:12:21.802664 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:21.803014 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:21.806471 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:21.816601 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:12:21.825353 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:21.826349 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:21.837260 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:21.847187 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:12:21.858122 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:21.883628 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:21.885934 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:22.023046 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.135231 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:12:22.267781 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:22.290066 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:22.295835 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:22.304298 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:22.304625 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:22.314183 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:22.329550 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:22.333108 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:22.341250 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.361033 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.367774 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.402763 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:22.416226 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.426652 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.465037 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.498406 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.559247 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.578501 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.680120 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:12:22.736528 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:22.748643 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.754359 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:12:22.794823 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:22.809039 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:22.826885 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.831018 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 97 similar messages]
I20240501 20:12:22.864495 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:22.864588 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.882376 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.889559 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:12:22.893200 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:22.914564 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:22.926643 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:22.935701 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:12:22.973158 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:22.988765 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:22.998813 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.000659 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:12:23.001355 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:23.031425 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.091996 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.108068 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:12:23.164157 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:23.165503 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.170524 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.180097 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:23.303359 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.346801 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.419924 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.426077 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.440199 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:23.474017 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:23.562043 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:23.592097 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:23.700968 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:12:23.722538 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:23.724201 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:12:23.742671 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:23.771770 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:23.771785 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:23.776693 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:23.827153 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:23.892330 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:23.925554 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.035795 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.072675 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.090795 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.091987 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:24.120520 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.147367 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.183588 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:24.195457 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:24.222128 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.232720 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.246214 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.254920 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:24.344599 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:12:24.361619 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:24.385798 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:24.405282 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:24.431975 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:24.433648 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:24.456624 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:24.482856 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:24.524742 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:12:24.600504 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:24.691758 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:24.707299 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.766439 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.840480 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.871177 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:24.883036 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:12:24.921530 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:24.921912 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:24.926841 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:12:24.936089 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:24.968415 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:12:25.004112 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:25.026610 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:25.027196 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:25.052013 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:25.072374 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.081316 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:12:25.115097 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.161280 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:12:25.176503 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:25.192016 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.192175 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.338163 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.383400 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.414887 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.469705 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.512621 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:25.597307 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.618921 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.660923 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.661659 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.674603 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.696802 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.699139 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.711005 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:25.720400 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.771660 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:12:25.820513 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:25.842689 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:25.864836 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:25.875370 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:25.875954 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:25.878439 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:25.920717 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:25.943622 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:25.998494 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:26.053682 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:26.094224 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:26.147265 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:26.207902 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:26.326242 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:26.353120 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:26.367062 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:26.398773 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:26.412184 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:26.436439 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:26.449647 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:12:26.452822 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:26.453040 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:26.453610 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:26.468569 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:26.491580 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:26.585891 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:26.591841 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:26.598590 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:26.630069 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:26.656387 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:26.661593 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:26.730834 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:26.772290 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:26.784593 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:26.870337 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:26.928167 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:26.930186 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:26.935647 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:12:26.955032 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:26.963518 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:26.964659 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:26.979876 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:27.028353 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:27.039767 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.070889 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:27.168675 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.302673 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:27.363457 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.393215 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.424916 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.438923 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.469926 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.480477 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:27.496491 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.629491 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.679989 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:27.719981 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:27.732571 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:12:27.736120 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:27.760609 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:27.788872 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:27.789795 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:27.809716 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:27.810380 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.844419 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.851418 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.859463 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240501 20:12:27.862797 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:27.863015 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240501 20:12:27.876509 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:27.878784 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:27.882803 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.905789 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.924961 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:27.974727 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:28.034250 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:28.057567 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:28.079789 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:28.112596 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:12:28.217911 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:28.246811 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:28.260391 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:28.325979 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:28.349870 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:28.354445 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:28.358940 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:28.360580 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:28.364146 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:28.411800 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:28.428172 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:12:28.438922 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:28.459813 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:28.461750 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:12:28.470446 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:28.477965 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:28.520467 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:28.539486 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:28.557957 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:28.559136 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:28.627867 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:28.768942 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:28.833680 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:28.851521 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:28.872807 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:12:28.908478 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:28.940865 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:28.942920 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:28.966794 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:28.984651 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:29.008298 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:29.039469 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:29.065048 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:29.067020 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:29.138532 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:29.154783 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:29.166426 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:29.207027 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:29.220986 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:29.301230 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:29.371099 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:29.412181 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:12:29.472910 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:29.542037 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:29.563506 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:29.668978 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:29.684585 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:29.698580 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:12:29.775775 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:29.787406 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:29.810566 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:29.819905 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:29.914858 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:29.916088 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:29.929396 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:29.948173 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:29.982478 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:29.984745 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.055672 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.156152 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.170892 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:30.183921 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.187577 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.188452 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.198894 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.244797 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.283958 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.319114 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:12:30.329527 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:30.400403 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:30.411811 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:30.415784 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:30.416720 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:30.436899 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:30.444555 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:30.490329 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.493062 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.497442 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:12:30.541337 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling FlushMRSOp(3a7e956aedc44f56b3bf629c80ff65e4): perf score=0.033385
W20240501 20:12:30.541404 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.560600 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.593432 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.610504 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:30.627861 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:12:30.628368 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: FlushMRSOp(3a7e956aedc44f56b3bf629c80ff65e4) complete. Timing: real 0.087s user 0.058s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8484,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":4268,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1892,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12246,"threads_started":2,"wal-append.queue_time_us":3266}
I20240501 20:12:30.632148 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling UndoDeltaBlockGCOp(3a7e956aedc44f56b3bf629c80ff65e4): 4581 bytes on disk
I20240501 20:12:30.633484 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: UndoDeltaBlockGCOp(3a7e956aedc44f56b3bf629c80ff65e4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 20:12:30.637881 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling FlushMRSOp(021fed14d5a140f3b42d359bae264f69): perf score=0.033354
W20240501 20:12:30.642467 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.646543 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:12:30.691462 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: FlushMRSOp(021fed14d5a140f3b42d359bae264f69) complete. Timing: real 0.053s user 0.029s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3558,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":6557,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":919,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":4035,"threads_started":2,"wal-append.queue_time_us":327}
I20240501 20:12:30.696962 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling UndoDeltaBlockGCOp(021fed14d5a140f3b42d359bae264f69): 4921 bytes on disk
I20240501 20:12:30.697557 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: UndoDeltaBlockGCOp(021fed14d5a140f3b42d359bae264f69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240501 20:12:30.711975 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.737910 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.792815 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:30.808956 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:30.840540 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:12:30.868176 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:30.879441 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:30.887339 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:30.904964 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:30.905927 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:30.910737 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:30.949682 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:30.963114 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:31.010068 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling FlushMRSOp(2b4384496d9048259bf353bd58a15008): perf score=0.033346
W20240501 20:12:31.025265 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:31.054912 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:31.100648 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: FlushMRSOp(2b4384496d9048259bf353bd58a15008) complete. Timing: real 0.090s user 0.038s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6437,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":5886,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2166,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12615,"threads_started":2,"wal-append.queue_time_us":311}
I20240501 20:12:31.103034 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling UndoDeltaBlockGCOp(2b4384496d9048259bf353bd58a15008): 4581 bytes on disk
I20240501 20:12:31.103672 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: UndoDeltaBlockGCOp(2b4384496d9048259bf353bd58a15008) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240501 20:12:31.122779 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:31.127363 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:31.331303 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:31.372839 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:31.428234 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:31.479979 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:31.500577 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:31.506525 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:31.551568 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:31.566956 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:12:31.579432 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:31.583613 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:31.583647 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:31.672551 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:31.673596 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:31.678412 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:31.692853 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:31.735880 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:31.749681 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:31.773044 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:31.829770 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:31.831375 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:31.968423 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:32.122351 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:32.147365 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:12:32.175374 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:32.192318 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:32.202117 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:32.243932 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:32.256652 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:32.268684 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:32.276401 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:32.290000 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:32.294299 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:12:32.294934 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:32.314774 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:32.316116 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:32.422776 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:32.452253 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:32.481838 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:32.555650 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:32.569113 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:32.590472 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:32.594516 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:32.594671 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:32.737075 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:32.744756 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:32.759640 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:12:32.787410 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:32.787714 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:32.811079 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:32.840567 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:32.845095 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:32.857867 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:32.870172 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:12:32.871392 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:32.871876 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:32.872397 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:32.923791 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:32.981163 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 20:12:33.006836 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:33.046124 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.055151 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.086251 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.088039 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.109284 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.125308 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 20:12:33.143801 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:33.152971 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.167124 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.240221 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.296141 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.304308 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:33.352540 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.352838 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:33.471797 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:33.480077 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240501 20:12:33.495898 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:33.496100 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:33.496951 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:33.536355 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:33.545648 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:33.557539 20902 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 46505.5us
W20240501 20:12:33.563371 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.564146 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:12:33.567212 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:33.584182 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:33.586328 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:33.592394 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:33.775306 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:33.795815 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:33.798336 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:33.826069 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:33.928943 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:33.948645 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:12:33.958001 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:33.969369 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:33.978561 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:33.986289 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:33.995772 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:34.003806 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:34.029702 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:34.071599 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:34.236739 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:34.272377 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:34.282749 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:34.306890 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:34.311275 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:34.317349 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:34.341785 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:34.341969 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:12:34.384920 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:34.478965 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:34.559706 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:34.562685 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:34.606775 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:34.611770 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:34.633411 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:34.637255 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:34.653388 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:12:34.707173 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:34.707630 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:34.889524 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:34.933547 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:34.934093 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:34.942787 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:34.948778 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:34.996734 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.000895 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.036103 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:35.103685 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.125324 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.253958 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:12:35.281981 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:35.304085 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.310384 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:12:35.318472 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:35.325234 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.333788 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:12:35.334295 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:35.349124 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:35.399928 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:35.428920 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:35.465940 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:35.486835 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:12:35.497102 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:35.540472 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.541509 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.549921 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.604961 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.778417 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.852734 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.859642 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.868252 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.873073 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:35.883052 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:35.898483 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:35.906643 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:35.923050 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:35.944770 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:35.948172 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:35.952345 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:35.963248 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:35.994894 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:12:36.005206 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:36.029678 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:36.065923 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:36.071488 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:36.105998 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:36.135952 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:36.186190 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:36.301291 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:36.307502 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:36.331178 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:36.401978 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:36.423959 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:36.423943 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:36.440896 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:36.442942 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:36.462685 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:36.468947 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:36.476220 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:36.492328 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:36.522891 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:36.531885 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:36.597131 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:36.768643 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:36.781028 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:36.850497 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:36.907085 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:36.985582 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.017851 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:37.143497 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:12:37.178406 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:37.184554 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:37.198874 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:37.201252 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:37.246351 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:37.248997 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:12:37.267302 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:37.268174 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:37.291561 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:37.302209 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:37.338557 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:37.395563 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.416844 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.426486 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.442546 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:37.555167 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.584988 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.636992 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.646312 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.673295 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:12:37.674072 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:37.691448 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:37.713917 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:37.714609 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:37.726286 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:37.733952 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:37.744823 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:37.749593 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:37.761159 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 20:12:37.769537 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.796068 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:37.798462 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:37.820542 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.824776 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:37.849407 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.895613 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:37.916785 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:12:37.959842 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:38.026974 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:38.148452 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:12:38.152393 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:38.216420 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:38.235641 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240501 20:12:38.299188 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:12:38.314430 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:38.326643 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:38.328354 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:38.339960 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:38.386528 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:38.387575 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:38.396986 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:38.405592 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:38.418763 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:38.457818 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:38.473333 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:38.487141 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:38.496898 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:38.531339 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:38.532745 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:38.568738 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:38.618906 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:38.686971 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:38.709554 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:38.717439 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:38.725916 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:38.782706 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:38.805150 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:38.806517 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:38.841820 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:38.939059 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:38.997679 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:39.002759 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:39.039345 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:12:39.040777 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:39.064167 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:39.083849 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:39.118358 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:39.124266 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 78576.6us
W20240501 20:12:39.124719 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 90733.5us
I20240501 20:12:39.135322 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:39.136713 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:12:39.221160 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:39.227999 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:39.241735 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:39.384614 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:39.406741 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:39.477828 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:39.538431 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:39.555773 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:12:39.571228 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:39.578557 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:39.596313 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:12:39.605852 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:39.617967 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:39.627403 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:39.667840 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:39.711330 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:39.730024 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:39.803319 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:39.873091 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:39.892086 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:39.964884 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:39.972549 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:40.050535 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:40.058214 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.090627 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:12:40.141209 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:40.147852 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:40.182581 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.202951 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:40.207321 23912 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 56933.2us
I20240501 20:12:40.240595 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:40.243184 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:40.263505 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:12:40.264015 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:40.279018 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.279588 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.290467 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.295411 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.300222 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:12:40.324048 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:40.355792 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:40.375298 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.457746 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.462445 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.580505 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:40.584856 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.650578 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.689916 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.732081 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.814854 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:40.837275 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.839267 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.850194 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.862634 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:40.869513 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:12:40.894614 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:40.924216 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:40.930794 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:40.947527 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:40.994943 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling FlushMRSOp(e5da8abdb3014753ac3cbef1d622069e): perf score=0.033400
W20240501 20:12:41.008162 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:41.010588 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:41.019219 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:41.020594 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:41.144809 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:41.176121 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:12:41.183626 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: FlushMRSOp(e5da8abdb3014753ac3cbef1d622069e) complete. Timing: real 0.188s user 0.052s sys 0.038s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":69401,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":3081,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":830,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":98442,"threads_started":2,"wal-append.queue_time_us":20082}
I20240501 20:12:41.186751 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling UndoDeltaBlockGCOp(e5da8abdb3014753ac3cbef1d622069e): 4834 bytes on disk
I20240501 20:12:41.188069 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: UndoDeltaBlockGCOp(e5da8abdb3014753ac3cbef1d622069e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240501 20:12:41.191890 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.195020 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.199676 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.254401 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.313925 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.354423 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.468554 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.476816 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:41.505939 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:41.535234 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:41.548583 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:41.560218 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.565703 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:12:41.658753 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:41.662791 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:41.684106 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:41.689774 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:41.721971 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.762638 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:41.772521 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:41.865602 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:41.892659 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:41.932104 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.021595 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:42.027069 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:42.048892 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:42.063127 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:42.067459 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:42.105482 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:42.157490 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:42.166987 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:42.187225 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:12:42.221251 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:42.252024 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.312191 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.417104 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:42.422026 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:42.441869 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.494572 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:42.501319 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.502046 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.527508 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.552569 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.590636 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.636677 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.691236 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.697690 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.757189 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.772933 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.787667 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:12:42.810839 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:42.816642 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:12:42.821146 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:42.838493 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:42.872952 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:42.909432 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:42.922734 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:42.926618 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:42.929526 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:42.965471 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:42.985643 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.005463 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.019656 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:43.182407 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.187932 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.198307 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:12:43.214608 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.247889 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.259164 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:43.280578 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240501 20:12:43.340996 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:43.353184 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:12:43.362430 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:43.384864 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:43.400990 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:12:43.406230 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:43.411145 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:43.441699 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:43.460399 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:43.483862 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:43.489744 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:43.498589 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.555444 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:43.590234 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:43.597359 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.658047 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.716863 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.778493 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:43.778962 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:43.789234 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:43.816660 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:43.826704 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:43.828457 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:43.830680 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:43.859983 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:43.861533 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:43.887979 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:43.903367 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:43.914497 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:43.914852 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:43.934736 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:43.961686 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:44.127682 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:44.184937 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:44.238137 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:44.263486 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:44.385154 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:44.391497 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:44.439497 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:44.497308 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:44.568477 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:44.589690 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:44.622996 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:44.646150 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:44.650468 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:12:44.653219 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:44.677156 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:44.684190 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:12:44.713361 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:12:44.742264 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:44.744874 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:44.805225 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:44.865027 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:44.939682 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:44.956775 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:44.974081 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:45.000855 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.010463 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.012485 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:45.012615 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.110767 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:45.111776 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:45.119155 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:45.124267 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.144407 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.161062 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:45.181567 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:45.199088 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:45.207990 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:45.209709 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:45.280156 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:45.296201 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.346925 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.378168 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:45.416011 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:45.495579 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.524863 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:45.604956 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:45.608543 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:45.622180 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:45.632779 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:45.641778 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:45.676978 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:45.680166 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:45.688822 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:45.703814 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.733153 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:45.742208 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:45.770596 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:45.783357 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:45.924970 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:45.928022 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:45.962028 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:45.974895 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:45.978878 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:46.040329 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:46.137120 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:46.185961 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:46.249369 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:46.292340 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:46.321593 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:46.339529 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:46.340700 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:46.348595 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:46.358922 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:46.387929 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:46.387970 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:46.400949 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:46.414132 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:46.441408 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:46.462316 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:46.490449 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:46.518924 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:46.529343 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:46.534844 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:46.569283 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:46.661650 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:46.801483 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:46.845105 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:46.859484 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:12:46.875311 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:46.895670 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:46.917833 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:46.921214 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:46.945405 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:46.967208 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:46.989320 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:47.015875 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:47.050976 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:47.135308 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:47.142181 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:12:47.212495 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:47.308706 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:47.345080 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling FlushMRSOp(10a4d24c5c844c8c9ccd191b1a2b0b00): perf score=0.033371
I20240501 20:12:47.394198 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: FlushMRSOp(10a4d24c5c844c8c9ccd191b1a2b0b00) complete. Timing: real 0.049s user 0.035s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":441,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":2557,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":759,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":767,"threads_started":2,"wal-append.queue_time_us":285}
I20240501 20:12:47.396494 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling UndoDeltaBlockGCOp(10a4d24c5c844c8c9ccd191b1a2b0b00): 4581 bytes on disk
W20240501 20:12:47.397843 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:47.399768 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:47.405964 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: UndoDeltaBlockGCOp(10a4d24c5c844c8c9ccd191b1a2b0b00) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
W20240501 20:12:47.415078 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:47.421160 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:47.429061 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:47.436029 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:47.449007 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:47.484856 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:47.493721 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:47.494729 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:47.495290 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:47.495857 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:47.526794 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:47.556613 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:47.566980 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:47.599037 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:47.629634 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:47.632579 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:47.657477 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:47.658990 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling FlushMRSOp(8d75b25758954877ab36e8029fb29463): perf score=0.033394
I20240501 20:12:47.703807 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: FlushMRSOp(8d75b25758954877ab36e8029fb29463) complete. Timing: real 0.045s user 0.031s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1590,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":896,"thread_start_us":255,"threads_started":1,"wal-append.queue_time_us":362}
I20240501 20:12:47.705600 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling UndoDeltaBlockGCOp(8d75b25758954877ab36e8029fb29463): 4835 bytes on disk
I20240501 20:12:47.706244 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: UndoDeltaBlockGCOp(8d75b25758954877ab36e8029fb29463) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240501 20:12:47.735486 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:47.777390 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling FlushMRSOp(748b4fd500f84a359bd8ee05b861300d): perf score=0.033363
W20240501 20:12:47.806691 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:47.817849 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: FlushMRSOp(748b4fd500f84a359bd8ee05b861300d) complete. Timing: real 0.040s user 0.033s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":322,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":952,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":411,"threads_started":2,"wal-append.queue_time_us":277}
I20240501 20:12:47.820190 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling UndoDeltaBlockGCOp(748b4fd500f84a359bd8ee05b861300d): 4834 bytes on disk
W20240501 20:12:47.820780 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
I20240501 20:12:47.820897 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: UndoDeltaBlockGCOp(748b4fd500f84a359bd8ee05b861300d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240501 20:12:47.850039 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:48.048475 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:48.052955 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:48.133126 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:48.139456 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:48.166105 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:48.200582 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:48.218889 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:48.237411 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:12:48.242369 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:48.245930 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:48.247051 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:48.281325 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:48.283004 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:48.311882 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:48.314509 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:48.332886 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 20:12:48.358987 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:48.365612 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:48.457966 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:48.459774 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:48.464885 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling FlushMRSOp(998f71aeae984fd5b866cb866bd4e87a): perf score=0.033393
W20240501 20:12:48.472612 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:48.486335 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:48.501996 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:12:48.505367 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:48.507959 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling FlushMRSOp(554b1d57dbb541508afaf8dbb77cbdad): perf score=0.033396
I20240501 20:12:48.527019 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: FlushMRSOp(998f71aeae984fd5b866cb866bd4e87a) complete. Timing: real 0.062s user 0.047s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6354,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":5028,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1075,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":12427,"threads_started":2,"wal-append.queue_time_us":6057}
I20240501 20:12:48.528928 23892 maintenance_manager.cc:419] P c6dcaa8efff642dcb7c2c6e8be3a65f7: Scheduling UndoDeltaBlockGCOp(998f71aeae984fd5b866cb866bd4e87a): 4834 bytes on disk
I20240501 20:12:48.529510 23783 maintenance_manager.cc:643] P c6dcaa8efff642dcb7c2c6e8be3a65f7: UndoDeltaBlockGCOp(998f71aeae984fd5b866cb866bd4e87a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240501 20:12:48.566478 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:12:48.599884 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: FlushMRSOp(554b1d57dbb541508afaf8dbb77cbdad) complete. Timing: real 0.076s user 0.048s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4890,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":4269,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":913,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5158,"threads_started":2,"wal-append.queue_time_us":304}
I20240501 20:12:48.602272 24023 maintenance_manager.cc:419] P 744c5a532e724391830c0785d32eb2fb: Scheduling UndoDeltaBlockGCOp(554b1d57dbb541508afaf8dbb77cbdad): 4921 bytes on disk
I20240501 20:12:48.602944 23913 maintenance_manager.cc:643] P 744c5a532e724391830c0785d32eb2fb: UndoDeltaBlockGCOp(554b1d57dbb541508afaf8dbb77cbdad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240501 20:12:48.613972 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:48.631636 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:48.640832 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:48.673496 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:48.709519 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:48.798662 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:48.808688 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:48.815184 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:48.823210 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:48.844107 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:48.848477 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:48.866770 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:48.887845 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:48.894951 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:48.914340 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:48.919219 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:48.970923 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:48.985213 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:49.020934 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:49.087934 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.175206 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:49.233686 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:49.273994 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:49.345741 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:49.352034 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:49.366920 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:49.367501 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:49.373672 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:49.401427 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:49.416409 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:49.423415 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:49.430258 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.451082 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:49.516801 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.553701 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:49.614887 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.733153 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:12:49.792650 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.851749 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:49.877852 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.879873 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.921224 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.924976 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.969384 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.970567 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:49.981822 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:50.022945 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:50.036598 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:50.066228 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:50.073880 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:50.084374 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:50.105549 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:50.124140 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:50.132052 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:50.132447 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:50.186580 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:50.195647 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:12:50.278106 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:50.280164 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:50.360625 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:50.439123 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:50.478776 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:50.538060 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:50.540227 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:50.543514 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:50.557165 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:50.568699 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:50.596860 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:50.614593 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:50.641095 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:50.668931 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:50.690356 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:50.706079 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:50.749222 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:50.773582 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:50.854161 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:50.950079 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:51.008652 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:51.029337 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:51.035017 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:51.063211 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:51.069365 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:51.078296 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:51.126272 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:51.141443 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:51.162359 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:51.193145 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:51.194065 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:51.209069 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:51.239303 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:51.269129 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:51.279171 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:51.302352 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:51.365605 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:51.377897 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:51.403085 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:51.454623 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:51.572974 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:51.648478 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:51.655807 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:51.710831 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:51.743930 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:51.803074 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:51.839131 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:51.882903 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:51.902081 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:51.902709 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:51.920143 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:51.921516 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:51.928694 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:51.976379 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:51.976454 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:51.996390 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:52.011518 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:52.016610 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:52.229771 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.230829 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:52.282761 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.292517 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.298861 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.301604 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:52.369706 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.376310 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.395898 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:12:52.412597 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:52.422066 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:52.425532 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:52.430055 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:52.430891 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:52.457244 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:52.460182 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.466182 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.468405 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:52.490875 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:52.496497 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.499413 23909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 57634.7us
I20240501 20:12:52.520002 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:52.540527 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:52.573889 23910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 124116us
W20240501 20:12:52.788053 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.810822 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.817703 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:52.869789 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:12:52.930439 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:52.930469 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:12:52.932694 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:12:52.936125 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:52.955480 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:52.965875 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:52.999356 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:53.002146 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:53.020066 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:53.047511 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:53.083298 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:12:53.085265 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.086973 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:53.091703 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:53.146718 23646 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 75112.4us
W20240501 20:12:53.169116 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:53.191079 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.219918 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.249855 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:12:53.342280 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 20:12:53.381464 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.427078 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.505983 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:12:53.531069 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:53.550334 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.592953 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:53.598480 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.618726 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:53.622840 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:12:53.623028 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:53.624145 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.667037 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:53.708060 23909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 55401.3us
W20240501 20:12:53.752935 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:53.766522 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:53.791323 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:53.792340 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:53.802783 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:53.817035 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:53.829224 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:53.830947 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:53.853669 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:53.853962 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:53.942391 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.949457 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:53.954329 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:53.986951 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:54.093250 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.226570 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.245163 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:54.271870 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:54.278587 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:54.297704 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:54.304092 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:54.327235 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:54.328317 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:54.355425 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:54.355453 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:54.383271 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.421202 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:54.474524 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.487828 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.510661 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.550514 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:54.600610 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.677223 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.706622 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.745536 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.756747 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:54.816704 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:54.818064 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:54.820111 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:54.839344 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:54.866745 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:54.891214 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:54.918808 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.935163 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:54.956199 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:54.957898 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:54.966107 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:54.966650 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:54.990779 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:12:55.107054 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:55.129849 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:55.144701 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:55.158078 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:55.410877 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:55.419251 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:55.476963 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:55.488309 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:55.488642 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:12:55.526163 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:55.600665 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:55.612068 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:55.629079 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:55.649940 23779 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 54888.3us
I20240501 20:12:55.674618 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:55.676816 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:55.682269 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:55.684381 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:55.699828 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:55.745795 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:55.754201 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:55.754907 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:55.778113 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:55.789599 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:55.937767 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:56.013803 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:56.028126 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:56.032714 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:56.080665 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:56.166456 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:56.178537 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:56.195840 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:56.213461 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:56.242785 23645 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 51186.3us
W20240501 20:12:56.254235 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 65702.6us
W20240501 20:12:56.270146 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:56.279901 23646 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 62195.9us
W20240501 20:12:56.303473 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:56.330017 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:56.362811 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:56.373539 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:56.392318 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:56.396934 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:56.446744 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:56.478307 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:56.483814 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:12:56.495862 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:56.562952 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:56.605803 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:56.703598 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:56.735011 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:56.742787 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:56.791983 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:56.806721 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:56.894593 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:56.912400 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:56.953095 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:56.955824 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:56.967161 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:56.968887 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.007800 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:57.010355 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:12:57.019871 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:57.049084 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:57.051278 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:57.052527 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:57.078892 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:57.095129 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.142071 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:57.193346 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.199402 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:57.260932 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:57.314697 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.349516 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.400732 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.472797 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:12:57.540549 20815 maintenance_manager.cc:419] P f878915f05c04c2c8c7cd9cf39cd6cf8: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033396
W20240501 20:12:57.567013 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:57.577126 20814 maintenance_manager.cc:643] P f878915f05c04c2c8c7cd9cf39cd6cf8: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.034s user 0.003s sys 0.004s Metrics: {"bytes_written":3892,"delete_count":0,"lbm_write_time_us":262,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":7296,"threads_started":1,"update_count":62,"wal-append.queue_time_us":4834}
W20240501 20:12:57.602136 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:57.614099 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.617004 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.667006 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.730012 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:57.796056 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:57.808110 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:57.809031 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:57.814100 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:57.839247 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:57.852753 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:57.867902 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:57.887626 23912 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 63290.2us
I20240501 20:12:57.894196 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:57.931037 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:12:57.932179 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:57.985859 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:58.023670 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:58.032847 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:58.075377 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:58.165561 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:58.181180 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:12:58.182094 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:58.191891 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:58.278764 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:12:58.280166 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:58.280411 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:58.303110 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:58.306661 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:58.314071 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:58.331655 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:58.334033 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:58.347103 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:58.356281 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:12:58.379568 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:12:58.394209 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:58.416095 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 20:12:58.427582 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:58.450749 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:58.510452 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:58.512802 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:58.545421 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:58.625600 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:12:58.636060 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:12:58.658592 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:58.766279 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:58.790601 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:58.811021 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:12:58.827107 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:12:58.835234 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:12:58.839241 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:58.839680 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:58.845109 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:12:58.849346 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:58.854481 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:58.854622 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:58.873610 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:59.025038 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:59.080111 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:59.191331 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling FlushMRSOp(75a1b78bb9574b588e6bf69ae7e68bc3): perf score=0.033403
W20240501 20:12:59.207079 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:59.216234 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:59.249668 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:59.266506 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: FlushMRSOp(75a1b78bb9574b588e6bf69ae7e68bc3) complete. Timing: real 0.075s user 0.056s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":673,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":3671,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":708,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":785,"threads_started":2,"wal-append.queue_time_us":333}
I20240501 20:12:59.268939 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling UndoDeltaBlockGCOp(75a1b78bb9574b588e6bf69ae7e68bc3): 4921 bytes on disk
I20240501 20:12:59.270263 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: UndoDeltaBlockGCOp(75a1b78bb9574b588e6bf69ae7e68bc3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240501 20:12:59.295766 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:59.312400 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:59.349772 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:59.398346 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:59.485272 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:59.499347 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:12:59.502420 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:12:59.519713 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:12:59.534550 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:12:59.537307 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:12:59.554162 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:12:59.554963 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:59.555087 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:59.567127 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:12:59.579242 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:12:59.598338 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:12:59.608686 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:12:59.615674 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:59.647898 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:12:59.658309 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:12:59.740617 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:12:59.748896 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:12:59.906325 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:12:59.962016 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.028386 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.046010 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:13:00.047888 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:00.070813 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:00.074585 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:00.117594 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:00.122148 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:00.125495 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:13:00.146538 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:00.160424 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:00.168665 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:00.184468 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.211397 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.250645 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.313725 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.373522 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:13:00.379462 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.455222 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:13:00.474591 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.542917 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.614792 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:00.622920 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:13:00.640254 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:00.655858 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:00.662514 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:13:00.663208 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:00.712128 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:00.716203 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:00.733294 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:00.765272 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:00.772857 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.799293 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.814072 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:13:00.818778 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:00.835862 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.879886 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:00.880136 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:00.907742 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:01.017042 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:01.089825 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:01.215705 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:01.299093 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:01.343742 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:01.353545 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:01.397804 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:01.422046 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:01.423441 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:01.446012 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:01.463799 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:01.482565 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:01.498250 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:01.539917 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:01.573066 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:01.637209 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:01.728683 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:01.750965 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:01.806869 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:01.841048 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:01.911792 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:01.914152 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:01.918211 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:01.934674 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:01.942306 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:01.954106 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:01.954618 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:01.955907 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:01.962167 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:01.964572 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:01.969159 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:02.053961 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:02.069815 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:02.102200 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.147378 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.241798 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.249562 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.281322 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.281462 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.331521 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.336791 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.373721 24045 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 49761.1us
W20240501 20:13:02.383922 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:13:02.456658 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:02.483273 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:02.484501 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:02.485625 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:02.493360 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:02.501981 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:02.514487 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:02.527810 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:02.528290 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:13:02.592492 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:02.596145 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.716177 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.751163 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.758354 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.885591 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.890188 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:02.934734 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:02.940953 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:13:02.978379 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:03.014885 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.076920 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:03.080839 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.157431 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.193966 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.218156 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:13:03.219218 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.252539 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:03.261447 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.314591 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.357707 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:13:03.383059 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:03.391912 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:03.398404 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:03.400323 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:03.404534 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:03.411924 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:03.417251 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:03.417937 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:03.423133 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:03.434617 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:03.487975 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:03.517242 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 20:13:03.535208 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.577744 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:03.676826 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.681169 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.726744 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240501 20:13:03.756394 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:03.757930 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:13:03.844751 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:03.868824 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:03.869467 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:03.890655 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:03.912175 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:03.947634 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:03.955740 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:03.962276 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:04.006453 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:04.014452 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:04.179970 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.218020 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:04.241278 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.260732 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:04.280517 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:04.287343 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:04.345858 24046 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 45943us
W20240501 20:13:04.358765 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.388845 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:04.407729 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:04.429608 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:04.446547 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:04.447669 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.473804 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:04.480281 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:04.491864 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:04.513967 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:04.521960 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:04.526728 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:04.546512 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:04.574925 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling FlushMRSOp(b70e144824074927bb0f4eafcdeadcec): perf score=0.033348
W20240501 20:13:04.580122 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.710273 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:04.718425 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: FlushMRSOp(b70e144824074927bb0f4eafcdeadcec) complete. Timing: real 0.143s user 0.050s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8497,"dirs.run_cpu_time_us":292,"dirs.run_wall_time_us":10268,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2645,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8874,"threads_started":2,"wal-append.queue_time_us":333}
I20240501 20:13:04.720333 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling UndoDeltaBlockGCOp(b70e144824074927bb0f4eafcdeadcec): 4581 bytes on disk
I20240501 20:13:04.720865 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: UndoDeltaBlockGCOp(b70e144824074927bb0f4eafcdeadcec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240501 20:13:04.749375 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.780781 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.831979 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.902297 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.915864 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.935374 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:04.969200 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:04.975615 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling FlushMRSOp(e1476e8f4c9343928f68365fd2772f0f): perf score=0.033344
I20240501 20:13:04.994164 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling FlushMRSOp(ca03186ff8b342dd958b205125b14177): perf score=0.033393
W20240501 20:13:05.017717 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:05.062932 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: FlushMRSOp(e1476e8f4c9343928f68365fd2772f0f) complete. Timing: real 0.087s user 0.046s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1830,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":654,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13263,"threads_started":1,"wal-append.queue_time_us":13371}
I20240501 20:13:05.065323 24156 maintenance_manager.cc:419] P 8559229e044b4e1ea661b881b3f9af50: Scheduling UndoDeltaBlockGCOp(e1476e8f4c9343928f68365fd2772f0f): 4919 bytes on disk
I20240501 20:13:05.066454 24047 maintenance_manager.cc:643] P 8559229e044b4e1ea661b881b3f9af50: UndoDeltaBlockGCOp(e1476e8f4c9343928f68365fd2772f0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240501 20:13:05.075102 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:05.094244 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: FlushMRSOp(ca03186ff8b342dd958b205125b14177) complete. Timing: real 0.100s user 0.061s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":370,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":883,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":879,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6666,"threads_started":2,"wal-append.queue_time_us":6278}
I20240501 20:13:05.096803 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling UndoDeltaBlockGCOp(ca03186ff8b342dd958b205125b14177): 4834 bytes on disk
I20240501 20:13:05.097359 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: UndoDeltaBlockGCOp(ca03186ff8b342dd958b205125b14177) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240501 20:13:05.114535 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:05.129575 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:05.243263 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:13:05.246927 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:05.324121 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:05.330806 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:05.338605 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:05.342919 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:13:05.350833 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:05.352604 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling FlushMRSOp(b4c4664b17f04e2d8285b7851b27d1a1): perf score=0.033396
I20240501 20:13:05.361052 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:05.392338 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:05.392742 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:05.429028 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:05.439483 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:05.477985 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:05.482630 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:05.547643 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: FlushMRSOp(b4c4664b17f04e2d8285b7851b27d1a1) complete. Timing: real 0.195s user 0.035s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1831,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1024,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":16867,"threads_started":1,"wal-append.queue_time_us":5991}
I20240501 20:13:05.550923 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling FlushMRSOp(24cc6ff1a0104ac690c8d8f82c17200c): perf score=0.033375
W20240501 20:13:05.572674 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:05.613001 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:05.664755 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:05.674294 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: FlushMRSOp(24cc6ff1a0104ac690c8d8f82c17200c) complete. Timing: real 0.116s user 0.060s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1410,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1149,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7320,"threads_started":1,"wal-append.queue_time_us":7409}
I20240501 20:13:05.676519 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling UndoDeltaBlockGCOp(b4c4664b17f04e2d8285b7851b27d1a1): 4921 bytes on disk
I20240501 20:13:05.677476 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: UndoDeltaBlockGCOp(b4c4664b17f04e2d8285b7851b27d1a1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 20:13:05.679726 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling UndoDeltaBlockGCOp(24cc6ff1a0104ac690c8d8f82c17200c): 4581 bytes on disk
I20240501 20:13:05.680383 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: UndoDeltaBlockGCOp(24cc6ff1a0104ac690c8d8f82c17200c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240501 20:13:05.711874 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:05.818879 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:05.831362 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:05.852156 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:05.899307 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:05.899634 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:05.920390 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:05.924335 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:05.953966 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:05.963069 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:05.975160 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:05.994161 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:06.046814 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:06.061949 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:06.068397 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:06.068516 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:06.076340 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:06.092684 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:06.121258 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:06.234498 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:06.271095 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:06.325203 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:06.502718 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:06.509948 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:06.519289 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:06.520689 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:06.522646 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:06.525800 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:06.535316 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:06.543462 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:06.577684 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:06.655426 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:06.655691 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:06.724025 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:06.741559 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:06.802677 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:06.815147 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:06.852818 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:06.914464 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:06.922278 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:06.972645 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:07.008899 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:07.035481 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.113080 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:07.137952 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:07.139837 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.209405 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:07.249919 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:07.260447 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:07.263584 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:07.282362 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:07.291034 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:07.298410 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.309592 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:07.334749 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 45939us
I20240501 20:13:07.343971 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:07.346079 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:07.419252 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:07.452415 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.476858 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.482154 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.553375 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:07.573765 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.676119 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.721592 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.761485 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:07.818719 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:07.822716 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:13:07.830001 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:07.865106 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:07.893465 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:07.904588 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:07.926476 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:07.928146 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:07.935319 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:07.941524 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:13:07.947463 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:07.953763 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:13:07.984143 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.048014 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.126030 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.154922 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:08.164698 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:08.219107 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.220968 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:08.252920 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:13:08.254132 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:08.275061 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.276679 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:08.360956 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:13:08.362744 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:08.424052 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:08.461041 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:08.485973 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:08.503645 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:08.540732 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.543140 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 20:13:08.545146 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:08.572625 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:08.575668 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.575795 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.621384 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:08.642514 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:08.651422 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:08.731220 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.742806 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:13:08.753015 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:08.828553 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:08.847496 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:13:08.848767 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:08.926607 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:08.968766 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:09.146132 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:09.218024 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:09.233675 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:09.247514 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:09.258534 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:09.330828 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:09.345886 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:09.359432 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:09.385172 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:09.402652 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:09.409835 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:09.458827 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:09.469873 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:09.485674 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:09.486318 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:09.506367 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:09.558763 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:09.625690 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:09.637562 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:09.703584 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:09.717029 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:09.737701 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:09.748898 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:09.841430 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:09.844888 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:09.846258 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:09.863974 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:09.865434 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:09.866824 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:09.883670 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:09.883939 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:09.884655 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:09.890868 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:09.940655 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:10.042078 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.084770 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.159651 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:10.162684 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.193781 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.247642 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.260489 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:13:10.312448 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:10.314545 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:10.315605 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:10.322829 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:10.334249 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:10.343308 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:10.362609 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:10.381045 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:10.381951 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:10.395253 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:13:10.497630 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.519651 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.536267 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:10.540028 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.617533 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.621905 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:10.766804 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:10.804388 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.813478 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.857869 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:10.873193 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:10.917923 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:10.946585 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:10.951400 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:10.959079 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:10.972589 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:11.026006 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:11.032814 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:11.057833 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:11.065449 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:11.067028 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 70428us
I20240501 20:13:11.069598 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:11.105768 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:11.133941 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:11.261533 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:11.333163 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:11.401754 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:11.434140 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:11.435061 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:11.473450 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:11.489517 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:11.517741 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:11.541173 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:11.542361 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:11.553000 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:11.553802 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:11.601087 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:11.615037 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:11.645241 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:11.760367 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:11.786963 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:11.798241 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:11.825758 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:11.858237 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:11.858376 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:11.909430 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:11.936976 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:12.031286 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:12.060691 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.070092 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:13:12.072214 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:12.076532 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:12.080962 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:12.112599 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:12.130554 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:12.139199 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:12.178579 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:12.198433 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:12.204133 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:12.258633 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:12.327307 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.371064 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.388638 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.436161 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:12.439142 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:12.456796 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.484339 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.648756 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.651913 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.717805 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.718032 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:12.834546 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:12.855880 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:12.869526 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:12.886142 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:12.898591 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:12.898897 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:12.903210 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:12.910029 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:12.913753 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:12.931563 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:12.947034 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:12.975658 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:12.993870 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:13.024801 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:13.039389 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:13:13.052735 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:13.064173 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:13.067255 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:13:13.263697 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:13.271054 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:13:13.272580 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:13.277657 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:13.362289 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:13.395117 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:13:13.407697 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:13.418053 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:13.430912 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:13.434916 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:13.438462 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:13.440390 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:13.465502 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:13.466701 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:13.470953 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:13.473970 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:13.489465 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:13.513193 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:13.594288 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:13.630863 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:13.668041 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 20:13:13.670281 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:13.753597 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:13.765726 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:13:13.793579 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:13.817963 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:13.940752 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:13.944957 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:13.950362 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:13.960973 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:13.973145 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:13.979638 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:13.985754 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:13:13.992771 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:14.044867 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:14.070734 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:14.162421 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:14.240761 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:14.317241 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:14.341524 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:14.357812 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:14.419528 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:14.448940 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:14.505056 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:14.514422 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:14.593897 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:14.650180 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 83353.4us
W20240501 20:13:14.659121 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:14.663785 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:14.668279 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:14.743628 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:14.745633 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:14.750770 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:14.774822 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:14.775362 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:14.790125 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:14.833279 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:14.833967 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:14.847047 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:14.847369 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:14.852361 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.033919 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.050825 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.058368 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.063030 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:15.165160 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:15.175774 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.181643 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.192406 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:15.251241 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:15.263074 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:15.277063 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:15.287068 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:15.294494 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:15.314435 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:15.331995 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:15.349113 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:15.356878 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.370663 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.479657 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:13:15.486886 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.502907 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:15.512449 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:15.532862 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.563560 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.651011 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:15.653134 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.669699 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.695003 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:15.786497 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:15.812906 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:15.814772 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:15.819422 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:15.835901 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:15.858817 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:15.863739 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:15.888024 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.889959 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:13:15.890120 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:15.943665 21033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 126298us
W20240501 20:13:15.948176 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:15.993743 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:16.005630 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:16.006048 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:16.084954 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:16.117218 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:16.121040 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:16.136312 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:16.145470 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:16.387323 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:16.483027 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:16.498639 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:16.511834 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:16.535424 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:16.564317 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:16.568830 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:16.589825 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:16.615787 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:16.643172 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:16.767566 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:16.774045 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:16.777940 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:16.808284 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:16.821334 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:16.866005 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:16.888917 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:16.909418 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:17.037492 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:17.041989 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:17.059963 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:17.073455 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:17.084465 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:17.092881 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:17.108189 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:17.116293 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:17.130551 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:17.139505 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:17.189680 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.243976 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.280089 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.361976 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.379503 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.389503 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.409844 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.440064 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:17.543099 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:17.547689 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:17.571264 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:17.584628 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:17.604154 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:17.636197 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:17.665409 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:17.671320 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:17.710012 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:17.733553 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.740767 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.750872 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.753904 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.818732 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:17.832334 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling FlushMRSOp(aa02aed57a7a4e6a95b149d2078dc265): perf score=0.033386
W20240501 20:13:17.862159 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:17.866075 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: FlushMRSOp(aa02aed57a7a4e6a95b149d2078dc265) complete. Timing: real 0.033s user 0.022s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":327,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1854,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":393,"threads_started":2,"wal-append.queue_time_us":265}
I20240501 20:13:17.868387 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling FlushMRSOp(ed9b47caf0a041fd962e4f18b0de5ff2): perf score=0.033363
W20240501 20:13:17.892858 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:13:17.901540 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:17.904647 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: FlushMRSOp(ed9b47caf0a041fd962e4f18b0de5ff2) complete. Timing: real 0.036s user 0.021s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":2104,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":652,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":504,"threads_started":1,"wal-append.queue_time_us":633}
I20240501 20:13:17.906585 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling UndoDeltaBlockGCOp(aa02aed57a7a4e6a95b149d2078dc265): 4834 bytes on disk
I20240501 20:13:17.907146 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: UndoDeltaBlockGCOp(aa02aed57a7a4e6a95b149d2078dc265) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 20:13:17.908883 23760 maintenance_manager.cc:419] P 51bf640e895b4e5eb91bdf7509ea7e46: Scheduling UndoDeltaBlockGCOp(ed9b47caf0a041fd962e4f18b0de5ff2): 4834 bytes on disk
I20240501 20:13:17.909317 23648 maintenance_manager.cc:643] P 51bf640e895b4e5eb91bdf7509ea7e46: UndoDeltaBlockGCOp(ed9b47caf0a041fd962e4f18b0de5ff2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240501 20:13:17.935993 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:17.940799 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.007298 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.050945 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.057677 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:18.119835 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.152891 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.171006 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:13:18.172338 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.197326 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:18.308849 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:18.316716 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:18.378540 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:13:18.378901 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:18.390713 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:18.394181 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:18.402263 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.402274 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:13:18.403291 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:18.411463 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:18.413297 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:18.420997 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:18.447139 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:18.476775 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:18.487255 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:13:18.506965 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:18.511483 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.594179 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:18.650151 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.668810 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 20:13:18.670532 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:18.863487 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:18.866663 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:18.873425 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:13:18.911556 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:18.923636 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:18.923521 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 66955.5us
I20240501 20:13:18.945765 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:18.946161 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:18.947930 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:18.955979 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:18.961175 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:19.008677 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:13:19.014950 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:19.092157 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:19.099200 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:19.140568 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:19.164871 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:19.165959 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:19.181828 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:19.191576 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:19.286554 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:19.361528 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:19.404279 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:19.534379 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:19.564487 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:19.596910 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:19.597033 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:19.602809 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:19.640774 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:19.691355 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:19.714421 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:19.731072 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:19.735163 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:19.752240 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:19.756078 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:19.778703 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:19.783625 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:19.803270 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:19.845048 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:19.915014 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:19.929248 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.189975 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.224596 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.300923 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:20.307092 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.307236 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.324839 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:13:20.379263 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:20.383987 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:20.409475 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:20.409636 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:20.426535 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:20.435315 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:20.446087 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:20.450891 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.456595 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:20.458956 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:20.464584 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:20.482954 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:20.484006 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:20.596081 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.622265 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.654973 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.657893 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:20.698993 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:20.722476 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.727581 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:20.771162 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.816942 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:20.845523 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:20.865916 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:20.886094 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:20.892220 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:20.902907 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:20.906231 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:20.912145 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:20.913308 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:20.937259 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:20.966430 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:13:21.070942 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:21.092216 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:21.093204 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:21.210369 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:21.264603 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:21.270087 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:21.321449 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:21.349129 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:21.395733 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:21.400000 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:21.406252 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:21.409274 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:21.429067 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:21.437039 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:21.438864 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:21.445525 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:21.553401 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:21.680989 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:21.682758 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:21.696257 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:21.761150 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:21.836709 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:21.869462 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:21.911936 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:21.930543 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:21.998591 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.047909 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:22.060473 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.166781 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:13:22.217267 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:22.221040 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:22.250891 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.251462 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.269534 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 61604.4us
W20240501 20:13:22.274591 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.275648 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:13:22.282660 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:22.291600 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:22.310775 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:22.329464 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 65434.9us
I20240501 20:13:22.333881 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:22.360599 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:22.365443 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:13:22.375224 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:22.433245 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.630869 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.637465 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.718967 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.779031 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:13:22.792433 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:22.792820 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:22.797698 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:22.826925 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:22.828955 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:22.848613 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:22.862614 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:22.883263 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:13:22.893621 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:22.900307 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:13:22.920610 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:22.921455 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.930308 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:22.939841 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:22.982029 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:22.997216 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.007477 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.015908 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.046978 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.212385 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.214637 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:23.259001 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:13:23.265633 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.268596 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:23.278183 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.309296 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:23.348788 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.354676 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.365262 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:13:23.381008 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:23.387751 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:23.403918 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:23.404945 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:23.405370 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:23.420583 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:13:23.427412 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:13:23.437538 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:23.440003 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:23.478780 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:23.482807 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.488276 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:23.559360 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:23.683641 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 20:13:23.820865 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:23.899616 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:23.902349 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:13:23.989610 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:24.029412 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:24.051625 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240501 20:13:24.066507 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:24.070267 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:24.104202 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:24.118949 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:24.131855 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:24.142767 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 58124.2us
I20240501 20:13:24.145438 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:24.166239 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.173748 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:24.182646 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:24.197899 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.212752 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:24.213143 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:24.237295 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.243225 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:24.310770 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.384434 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.422729 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.437875 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:13:24.573035 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:24.580080 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:13:24.580845 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:24.588558 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:24.600445 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:24.648917 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:24.668465 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:24.699446 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.722769 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:24.723635 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:24.747103 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:24.785708 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.832640 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.863034 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:24.872112 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:24.912233 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.056245 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:25.074813 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:25.079607 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:25.086911 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:25.087419 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:25.112375 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:25.135208 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.181946 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:25.202706 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:25.214303 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:25.240633 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:25.249567 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:25.251856 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.285274 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:13:25.337216 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.355229 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:25.424029 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.436156 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:25.508524 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.540748 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.668404 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:25.676290 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.704365 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:13:25.758414 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.768286 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.776190 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:13:25.836314 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:25.842670 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:25.870870 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:25.881675 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:25.922050 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:25.937417 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:25.938462 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:25.940760 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:25.954087 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:25.965057 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:25.974941 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:25.975978 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:25.987932 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:25.990013 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:26.006757 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:26.010107 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:26.027215 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:26.056121 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:26.199352 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:26.206272 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:26.428925 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:26.444581 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:26.447834 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:26.453642 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:26.459501 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:26.497473 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:26.501739 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:26.509752 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:26.522382 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:26.560834 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:26.702035 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:26.714722 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:26.782837 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:26.803407 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:26.803928 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:26.808976 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:26.852639 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:26.889988 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:26.984171 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:26.987471 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:26.994232 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:27.001264 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:27.030611 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:27.094305 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:27.137159 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:27.142009 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:27.147624 20901 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 44261us
W20240501 20:13:27.167912 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:27.167912 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:27.172734 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:27.184201 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:27.264955 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:27.366595 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:27.375146 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:27.447999 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:27.481281 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:27.634449 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:27.675225 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:27.682242 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:27.745744 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:27.760320 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:27.805353 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:27.818799 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:13:27.854918 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:27.866819 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:27.883697 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:27.891636 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:27.892499 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:27.896162 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:27.926738 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:27.931061 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:27.934433 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:27.935343 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.014626 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.048481 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:28.138113 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.174976 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.190389 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.191664 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:28.236884 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:28.252480 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.291896 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:13:28.296015 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:28.311762 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:28.335181 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:28.336323 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:28.356287 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:13:28.372084 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:28.383270 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:28.383893 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:28.407276 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:28.416481 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:28.421955 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:13:28.431829 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:28.435906 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:28.438588 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:28.440994 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:28.445201 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:28.448139 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:13:28.459828 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.717805 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.741510 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 20:13:28.758375 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.784114 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:28.806488 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:28.826192 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.826362 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:28.845662 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:28.846129 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:28.858311 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:28.870509 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:28.877293 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:28.878897 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:28.896008 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:28.914816 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:13:28.915761 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:28.924656 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:29.042227 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.134562 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.168545 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:29.199254 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.204294 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:13:29.233162 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.267513 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:29.284778 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.324213 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.390552 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:29.428321 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.442472 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:29.476696 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:29.488639 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:29.575353 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:29.575490 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:29.581022 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:29.602352 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:29.629778 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:29.660499 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:29.661353 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:29.678824 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.812930 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.929144 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.971069 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:29.984170 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.002467 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.028743 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:30.034936 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:13:30.136209 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:30.153175 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:30.186995 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:30.198487 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:30.251960 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:30.278512 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.297396 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:30.307051 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:30.329198 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:30.352612 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:30.407439 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.420923 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.421695 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.438854 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:30.527886 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.545902 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.624531 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.656381 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.731215 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:30.750653 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:30.779141 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:30.795708 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:30.822656 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:30.843858 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:30.862841 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:30.865989 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:30.876160 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:30.886862 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:30.897481 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:30.917651 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:30.928306 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:30.936152 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:30.939254 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:30.958158 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:31.026165 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.026188 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.030244 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.032255 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:13:31.059409 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:31.343135 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:31.366025 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:31.393381 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.402513 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.456713 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:31.472088 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.499783 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:31.531245 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:31.547889 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:31.562559 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:31.565455 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:31.571084 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:31.584898 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:31.615872 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:31.629948 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:31.646979 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:31.653932 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:31.682904 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.688670 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.709863 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:31.806237 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.844524 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:31.846400 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:31.852679 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:31.857685 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:31.948017 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:31.950732 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:32.044117 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:32.059005 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:32.062856 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:32.110531 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:32.197050 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:32.198526 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:32.218909 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:32.219211 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:32.244937 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:32.273985 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 88670.4us
I20240501 20:13:32.274655 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:32.319324 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:32.323913 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:32.397328 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:32.634791 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:32.657579 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:32.702858 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:32.705291 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:13:32.722299 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:32.724043 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:32.757598 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:32.790727 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:32.794555 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:32.807583 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:32.829926 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:13:32.842514 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:32.892262 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:32.898806 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:32.925068 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:32.986325 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:33.007833 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.028972 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.068845 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.095204 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:33.251722 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.278093 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:33.297031 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:13:33.320823 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.341034 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.356037 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:33.453455 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.458760 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:13:33.460228 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 20:13:33.461097 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:33.461230 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.471709 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.476377 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:33.496531 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:33.506707 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:33.523790 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:33.546520 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:33.549647 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:33.589677 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:33.652384 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:33.660049 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:33.702411 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:33.720902 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.721958 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 54619us
W20240501 20:13:33.752113 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240501 20:13:33.808424 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.821782 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:33.821974 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:33.830405 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:33.888249 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.919871 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:13:33.945987 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:33.946826 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:33.984481 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:34.015928 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:34.042874 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:34.064762 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:34.076618 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:34.081568 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:34.099797 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:34.103663 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:34.163771 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:34.164870 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:34.180677 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:34.222594 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:13:34.242806 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:34.254906 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:34.256487 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:34.323483 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:34.344378 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:34.369453 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:34.370445 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:34.419660 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:34.454450 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:34.632058 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:34.641594 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:34.655936 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:34.669971 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:34.671213 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:34.671680 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:34.687803 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:34.753417 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:34.768576 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:34.775043 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:34.788487 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:34.807710 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:34.924217 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:13:34.958788 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:34.967317 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:35.086120 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:35.297968 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 47202.6us
W20240501 20:13:35.338497 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:35.387028 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:35.396477 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:35.419143 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:35.440024 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:35.445838 23910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 50520.8us
W20240501 20:13:35.459811 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:35.462380 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:35.472762 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:35.476401 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:35.480402 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:35.485460 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:35.486933 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:35.527805 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:35.571681 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:13:35.578486 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:35.583765 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:35.596899 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:35.647639 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:35.658926 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:35.769767 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:35.776882 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:35.860201 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:35.917104 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:35.948715 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:35.981824 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:35.992055 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:36.002929 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:36.016104 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:36.021580 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:36.030673 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:36.039875 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:13:36.043735 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:36.051234 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:36.056655 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:36.076699 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:36.143623 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:36.211402 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:36.252667 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:36.255069 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:36.301348 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:36.305222 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:36.326480 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:36.359418 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:36.363276 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:36.383369 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:36.498304 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:36.532441 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:36.542477 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:36.553879 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:36.556120 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:36.563230 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:36.579205 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:36.579455 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:36.601470 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:36.619247 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:36.636991 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:36.693708 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:36.723711 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:36.725416 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:36.771915 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:36.836241 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:36.884306 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:36.924322 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.024236 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.106997 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.154457 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:37.156136 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.163264 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.325127 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:13:37.341339 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:37.349834 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:37.350399 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:37.356783 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:37.380443 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:37.397677 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:37.461040 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:37.473265 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:13:37.491405 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.573832 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:37.623440 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.653951 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.751130 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.761229 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:37.797295 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:37.831804 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:37.841017 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:13:37.846534 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:37.847349 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:37.879990 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:37.880421 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:37.895768 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:37.909718 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:37.920590 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.033084 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.153126 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:38.160626 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.180754 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.193563 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:38.214001 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:38.241817 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:38.259636 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:38.261448 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:13:38.324736 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:38.335191 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:38.335767 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.336230 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:13:38.343148 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:13:38.367929 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:38.372946 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:38.378475 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.394934 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:38.397671 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:38.399497 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:38.421907 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:38.422479 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:38.440591 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:38.468127 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:13:38.473070 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:13:38.518143 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.580875 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.714684 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.715704 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.733289 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.734802 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:38.779078 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.806146 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 20:13:38.807514 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:38.840593 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.874243 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:38.899495 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:38.918282 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:38.988157 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:13:39.012099 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:39.012629 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:13:39.026784 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:39.039103 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:13:39.048897 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:13:39.052194 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:39.095119 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:39.115722 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:39.148965 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:39.149509 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:39.222520 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:39.228634 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:39.264477 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:39.286470 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:39.310408 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:13:39.377564 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:39.481271 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:39.537943 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:39.588963 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:39.623827 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:39.649740 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:39.653967 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:39.660693 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:39.662329 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:39.674618 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:39.683665 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 49532.4us
W20240501 20:13:39.703086 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:39.717415 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:39.731666 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 47705.8us
I20240501 20:13:39.755182 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:39.759850 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:39.765576 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:39.803975 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:39.896864 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:13:39.959762 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:39.992035 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:40.006891 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:40.079743 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:40.114487 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:40.251462 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:40.264843 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:40.293746 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:40.305984 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:40.307315 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:40.307596 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:40.321549 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:40.370894 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:40.378791 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:13:40.382706 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:40.383105 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:40.392364 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:40.668778 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:40.688014 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:40.694387 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:40.715916 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:40.719616 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:40.743099 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:40.832861 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:40.837971 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:40.843112 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:40.862283 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:13:40.937619 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:40.945278 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:40.981549 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:41.004168 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:41.007786 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:41.013746 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:41.026453 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:41.046546 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:41.095523 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:41.122752 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:41.136754 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:41.197851 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:41.205792 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:41.230417 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:41.233232 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:41.263221 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:41.331892 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:41.332815 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:41.345122 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:41.381397 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:41.403635 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:41.426553 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:41.444381 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:41.448546 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:41.515988 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:41.518747 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:41.522058 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:41.535173 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:41.542645 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:41.583598 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:41.583910 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:41.590170 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:41.656333 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:41.675261 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:41.678941 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:41.736354 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:41.826179 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:13:41.988407 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:41.991263 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:41.992113 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:42.023875 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:42.033270 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:42.033833 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:42.034111 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:42.038018 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:42.084678 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:42.109280 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.138142 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.155506 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.223786 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.237331 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.392421 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.411906 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:42.454488 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.504498 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.550930 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.556620 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:42.587271 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.619513 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.624469 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:13:42.636703 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:13:42.761170 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:42.767382 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:13:42.781759 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:42.804019 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:42.820541 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:42.822268 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:42.823788 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:42.851739 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:42.974242 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.984578 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:42.993590 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:42.998790 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.032680 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:43.058758 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:43.071826 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:43.223203 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.259243 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.266829 23646 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 55162.2us
I20240501 20:13:43.393319 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:43.408972 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
I20240501 20:13:43.421710 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:43.424961 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:13:43.456065 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:43.479782 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:43.482198 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:43.494983 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
I20240501 20:13:43.501715 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:43.514223 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.514798 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:13:43.531272 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:43.531468 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.532246 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:43.533550 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:43.538240 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.587982 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:43.593751 23909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 47108.2us
W20240501 20:13:43.639652 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:13:43.656980 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:43.697513 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:43.704283 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.714355 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:43.773571 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.808218 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 20:13:43.809650 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.948269 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:13:43.953181 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:43.956470 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:43.962098 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:43.979830 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:43.986354 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:43.996019 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:43.998921 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:44.005029 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:44.021497 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:13:44.037150 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:44.053572 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:13:44.073657 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:44.124878 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:44.125694 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:44.125869 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:44.157961 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 43744.4us
W20240501 20:13:44.192267 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:13:44.206044 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:44.218926 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:13:44.238812 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:44.438576 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:13:44.457141 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:44.471971 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:44.503401 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:44.629643 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:44.663993 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:44.689210 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:13:44.822726 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:44.822968 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:44.837872 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:44.843587 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:13:44.859230 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:44.888507 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:44.890201 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:44.891346 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:44.898056 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:44.928396 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:44.956199 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:44.966239 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:44.996052 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.004588 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.125489 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:13:45.127678 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.129782 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.257395 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.264094 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:13:45.294131 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:45.304155 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:45.334959 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:45.344118 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:45.349076 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.384799 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:13:45.413705 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:45.437973 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:45.448285 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:45.464881 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:45.479789 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:45.492213 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:45.600226 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.799927 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.827778 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:13:45.865520 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:45.871769 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:13:45.872830 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:45.875598 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.884249 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:45.888476 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:13:45.894058 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:45.900141 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:45.912755 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:45.915968 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:45.939843 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:45.973237 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:45.971424 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:46.095021 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:46.125631 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.144790 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:46.145550 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.222960 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:46.258041 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.287104 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.294068 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:46.307165 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:46.398885 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.478550 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.500519 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.541555 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:46.570362 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:13:46.572999 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:46.577677 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:46.579402 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:46.634967 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:46.645591 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:46.692317 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:46.700592 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.729780 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:46.740828 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:46.747924 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:46.762012 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:46.772122 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:46.820302 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.841881 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:46.917907 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:46.956995 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:47.129900 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:47.154516 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:47.169692 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:47.185822 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:47.198560 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:47.225765 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:47.232556 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:47.262095 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:47.301957 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:47.306393 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:13:47.306803 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:47.312515 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:47.319957 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:47.332209 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:47.407352 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:47.447671 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:47.588232 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:47.611047 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:47.622323 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:13:47.640481 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:47.717989 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:47.720320 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:47.735522 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:47.738225 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:47.746140 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:47.750224 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:47.786531 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:47.796686 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:47.850948 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.008344 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.036101 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:48.059178 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:48.061684 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.119431 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.123571 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.144074 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.149114 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.262234 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.286542 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:48.393956 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:48.422984 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:48.438423 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:13:48.445650 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:48.479364 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:48.483237 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.525326 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 20:13:48.533571 23909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 109536us
I20240501 20:13:48.539090 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:48.549779 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:48.557482 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:48.560953 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:48.567241 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:48.575352 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:48.591925 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:48.600061 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:48.615535 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.631337 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:48.675770 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:13:48.690902 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:48.702175 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.703583 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:48.765640 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.797472 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:48.854467 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 20:13:48.889192 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:48.939975 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:48.950249 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:13:48.954849 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:48.962684 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:48.966496 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:48.994205 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:48.996965 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:49.008277 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:49.020352 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:49.025065 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:49.048082 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:49.066532 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:13:49.070212 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:49.076126 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:49.142637 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.158573 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.165779 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:49.185652 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:49.241636 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.362330 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.405884 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.443141 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.486620 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.507454 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:13:49.536548 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:49.537822 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:49.566905 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:49.619575 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:49.620523 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:49.623097 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:49.631700 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:49.653999 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:49.688705 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:49.689561 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:49.746580 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:13:49.747447 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.786738 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.854068 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:49.991828 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:50.039458 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:50.132903 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.143811 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.253062 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:50.314327 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:50.327234 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:50.329291 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:50.347347 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:50.397068 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:50.425757 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.434756 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:50.454401 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:50.455112 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:50.455211 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:50.524067 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.531832 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.551121 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.579196 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.604691 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:50.733789 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.790287 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.852540 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:50.862972 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:50.866349 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:50.885268 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:50.886940 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:50.903439 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:50.909308 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:50.926335 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:50.935644 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:50.948741 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:50.991760 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.019843 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.060675 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:51.065131 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.123572 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:51.138458 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:51.209209 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:51.220067 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:51.222222 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:51.282908 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:51.288138 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:51.321566 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.350617 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:51.361994 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:51.370520 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:51.372478 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:51.415325 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:13:51.436657 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:51.445832 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:51.476912 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:51.510581 20901 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 82034.9us
W20240501 20:13:51.520047 20902 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 95039.6us
I20240501 20:13:51.524811 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:51.529708 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:51.542814 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:51.555876 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:51.571579 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:51.591898 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.608865 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.646871 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:13:51.658020 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.662886 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:51.736831 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.751322 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.770445 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.872359 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:51.935918 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:52.020370 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.098264 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.107651 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:52.109516 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:13:52.276727 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:52.281052 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:52.281844 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:52.288398 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.311565 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:52.313654 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:52.331652 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:52.336946 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:52.354384 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:52.356688 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:52.379709 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.435230 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:52.534600 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.548614 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:52.727218 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.753386 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.761348 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:13:52.818739 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:52.823797 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:52.835824 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:52.835825 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:52.847844 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:52.854740 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:52.877599 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:52.898643 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:52.921542 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.955592 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.970345 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:52.993549 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.061573 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:53.203758 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:13:53.314678 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:53.324576 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:13:53.326539 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:53.327702 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:53.335724 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:53.348932 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:53.386301 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:53.393182 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:53.428423 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:13:53.432844 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:53.437422 23912 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 85201.6us
W20240501 20:13:53.445037 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:13:53.460533 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.470084 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:53.478621 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.488912 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:53.534936 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:13:53.549993 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.620221 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.653347 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.704401 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:13:53.711711 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.721463 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.751796 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:53.785643 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:53.850322 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.856752 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.860836 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:53.862702 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:53.896374 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:53.944597 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 20:13:53.945698 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:54.006785 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:54.006922 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:13:54.011126 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:54.038481 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:54.050247 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:54.060494 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:54.087064 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:13:54.103152 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:54.105089 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:54.123917 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:54.132129 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:54.142341 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:13:54.150051 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:54.183894 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:13:54.185401 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:54.195756 23781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.136:44267, user_credentials={real_user=slave}} blocked reactor thread for 47700.4us
I20240501 20:13:54.205137 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:54.289223 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:54.398377 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:54.483676 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:54.532442 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:13:54.582978 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:54.592718 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:54.608080 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:13:54.619825 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:54.624693 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:13:54.650453 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:54.676599 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:54.680548 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:54.696641 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:54.704234 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:54.710539 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:54.726246 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:54.802459 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:54.824371 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:54.849939 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:54.936120 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:54.945070 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:55.004070 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:55.028687 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:13:55.041513 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:55.055241 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:55.130918 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:55.155325 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:55.192075 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:55.204929 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:55.232589 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:13:55.278299 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:55.284379 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:13:55.300257 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:55.307341 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:13:55.307799 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:55.405870 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:55.611889 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:55.656525 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:55.686259 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:55.701884 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:55.862732 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:55.877310 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:55.890125 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:55.905346 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:55.912348 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:55.928203 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:55.933830 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:55.953641 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:55.961869 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:56.009912 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.052292 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.109925 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.118508 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.119448 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.122901 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.171521 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:56.177311 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:56.244235 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.249753 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.280814 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.297111 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:56.311898 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:56.318953 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.333000 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.343744 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:13:56.368412 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:56.372234 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:56.380156 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.386996 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:56.387234 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:56.389899 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:56.396212 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:56.462893 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:56.466732 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:56.492251 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:56.529253 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:56.542646 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:56.551719 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:56.608928 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:56.673823 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.753470 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:56.845402 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:13:56.850524 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:56.852218 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:56.855350 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:56.889411 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:56.907481 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:56.927001 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:56.928951 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:56.948673 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:56.953539 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:56.984483 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:56.991317 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:13:57.074002 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:57.182950 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:57.202224 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:57.251955 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:57.416648 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:57.418848 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:57.442648 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:57.451464 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:57.481823 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:57.518252 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:57.561981 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:13:57.627139 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:57.675873 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:13:57.679356 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:13:57.684118 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:57.725852 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:57.757112 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:57.770643 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:57.771400 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:13:57.811717 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:57.864185 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:57.883941 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:57.972575 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:58.075846 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:58.147843 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:58.211208 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:58.258612 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:58.263783 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:58.274704 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:58.277079 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:58.299886 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:13:58.319031 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:58.328606 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 63113.9us
W20240501 20:13:58.329378 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:58.332489 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:58.366917 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:58.378491 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:13:58.409307 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:58.497205 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:58.505232 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:13:58.514197 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:58.538486 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:13:58.604233 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:58.608386 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:58.614073 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:58.650647 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:58.672464 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:58.688294 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:58.727412 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:58.769858 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:58.791973 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:58.811003 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:58.817040 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:58.817499 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:13:58.827361 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:58.846637 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:13:58.849301 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:58.859781 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:58.874464 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:58.880213 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:13:58.902050 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:13:58.937912 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:13:58.939092 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:58.949499 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:58.995374 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:59.003844 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240501 20:13:59.005007 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:59.034229 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:59.042886 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:59.090060 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:59.169332 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240501 20:13:59.253795 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:59.293959 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:59.361922 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:59.375336 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:59.392940 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:59.429821 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:13:59.456146 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:59.474062 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:13:59.479632 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:13:59.489288 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:13:59.493422 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:13:59.494204 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:13:59.504201 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:13:59.504258 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:13:59.519186 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:13:59.552827 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:13:59.579984 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:59.624275 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:59.660502 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:13:59.678804 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:13:59.830430 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:59.876633 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:13:59.904466 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:13:59.985507 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:00.019835 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:00.035621 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:00.055680 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:00.063803 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:00.076990 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:00.089623 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:14:00.132606 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:00.155386 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:00.161900 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:14:00.203521 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:00.229388 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:00.381150 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:00.420652 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:00.494040 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:00.510496 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:00.560245 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:00.560518 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:00.562150 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:00.572597 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:00.574761 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:00.621881 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:00.633206 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:00.640880 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:00.664777 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:00.703050 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:14:00.719135 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:00.719341 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:00.730404 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:00.751184 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:00.822609 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:00.858879 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:00.896147 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:00.970484 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:00.980553 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.019515 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:01.034013 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:14:01.118984 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.194118 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.228952 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:01.310654 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:01.332998 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:01.353395 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:01.354667 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:01.356916 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:01.379475 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:01.381862 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:01.386171 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:01.394915 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:01.436623 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:14:01.446688 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:01.453881 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.522728 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.564805 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:01.570487 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.575412 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:01.592409 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.635504 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:14:01.798028 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:01.813261 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:01.821741 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:01.821723 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:01.827584 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:01.829735 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:14:01.846215 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:01.850250 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:01.878260 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:01.879441 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.918299 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.943358 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:14:01.967983 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:01.999825 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:02.013161 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.022794 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.026329 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.091229 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.144933 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:02.285708 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.290409 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.317477 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.343000 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:02.345306 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:02.346088 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:02.348945 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:02.389324 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:02.418542 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:02.439455 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:02.452427 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:02.452643 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:14:02.456560 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:02.573710 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.579675 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:02.758165 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.838910 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.892184 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:02.973203 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.006186 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:03.057763 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:03.063413 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:03.109853 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:14:03.130151 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:03.131011 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:03.131603 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:14:03.140182 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:03.152782 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:03.152909 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:14:03.173106 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:03.173592 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:03.175141 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:03.195003 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:03.219666 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:03.224323 23912 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 50657.2us
W20240501 20:14:03.225597 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.268788 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.370862 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.465885 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.503634 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:03.544795 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:14:03.567257 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:14:03.569916 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.570272 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:03.588100 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:03.590519 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.595872 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.614807 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:03.618849 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:03.649873 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:03.668828 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:03.683030 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:03.733659 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:03.760967 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:03.825428 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:03.836061 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:03.936498 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.951462 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:03.953956 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:04.009188 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:04.013074 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:04.018548 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 20:14:04.020484 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:04.124989 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:04.153930 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:04.165370 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:04.179652 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:04.185190 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:14:04.254575 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:04.257354 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:14:04.281309 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:04.288625 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:14:04.314390 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:14:04.329043 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:04.330150 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:04.383206 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:04.388002 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:04.392675 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:14:04.402062 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:04.406656 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:04.465173 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:04.492539 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:04.547087 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:04.558662 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:04.565984 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:04.662233 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:04.700107 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:04.703898 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:04.807233 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:04.813146 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:04.865181 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:04.965814 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:04.981374 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:04.984370 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:04.991106 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:04.997825 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:04.999287 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:05.025086 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:05.033161 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:05.056512 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:05.057051 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.096586 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:05.097079 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.220131 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.358414 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.410255 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:05.438977 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.448361 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.473860 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:05.521817 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:05.531291 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:05.535871 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:05.553159 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:05.625998 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:14:05.628324 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:05.645366 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:05.668802 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:05.693933 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:05.811764 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.826016 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:05.835749 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.839268 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.839358 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.840258 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:05.853211 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:05.927577 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:05.935856 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:05.951326 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:05.979831 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:06.011195 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:06.016265 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:06.019629 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:06.023775 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:06.045265 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:06.074374 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:06.077536 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:06.078981 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:06.104725 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:06.201956 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:06.206403 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:06.324784 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:06.372052 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:06.444496 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:06.466646 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:06.580483 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:06.588678 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:06.609050 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:06.614105 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:06.621099 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:06.622633 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:06.701829 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:14:06.730218 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:06.814924 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:06.834128 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:06.848922 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:06.854267 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:06.865581 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:06.897536 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:06.899610 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:06.943423 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:06.946139 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:06.951054 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:06.969602 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:07.047427 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:07.085124 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:07.108142 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:07.112962 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:07.131587 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:07.183629 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:07.207275 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:07.264526 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:07.358407 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:07.400938 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:07.434865 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:07.439172 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:07.467656 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:14:07.513598 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:07.535377 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:07.536994 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:07.544656 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:14:07.549758 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:07.562775 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:07.635896 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:07.644484 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:07.732106 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:07.792928 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:07.879875 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:07.879900 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:14:07.930536 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:07.940253 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:07.973428 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:07.974469 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:07.993806 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:08.017911 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:08.018923 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:08.090904 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.095469 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.138818 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.138947 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.149518 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.154351 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.154474 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.253476 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:14:08.291555 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.323094 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:08.345436 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:08.368672 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:08.394482 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:08.567939 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 20:14:08.568127 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:08.594449 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:14:08.613034 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.630116 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 69829us
W20240501 20:14:08.647241 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:08.738037 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.741676 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:14:08.751664 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:08.804847 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:08.814257 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:08.824501 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:08.833909 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:08.895891 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.896225 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.922302 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:14:08.923117 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:08.940325 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:08.942234 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:08.995831 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:09.013571 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:09.044224 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:09.066430 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:09.075146 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 20:14:09.102690 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:09.103506 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:09.200604 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:14:09.201448 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:09.225772 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:09.311414 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:09.346817 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:09.377887 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:14:09.421455 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:09.422713 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:09.424275 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:09.427222 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:09.436878 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:09.460198 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:09.464609 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:09.470268 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:09.475898 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:09.508407 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:14:09.508430 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:09.541985 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:09.547346 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:09.600579 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:09.641551 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:09.648628 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:09.710559 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:14:09.711364 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:09.831909 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:09.836169 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:09.911800 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:09.961171 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:09.969880 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:09.972309 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:09.979573 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:09.997009 21033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 58745.5us
W20240501 20:14:10.000871 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:10.002110 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:10.060642 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:10.068439 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:10.092079 24043 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 46638.9us
W20240501 20:14:10.125967 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:10.148872 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 150096us
W20240501 20:14:10.160220 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:10.268352 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:10.299034 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:10.431146 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:10.457916 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:10.485360 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:10.493773 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:10.498327 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:10.565207 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:10.668992 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:10.679327 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:10.687479 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:10.749449 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:10.766353 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:10.767753 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:10.781566 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:10.795506 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:10.825789 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:10.830098 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:10.843187 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:10.845046 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:10.866922 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:14:10.871855 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:10.877979 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:10.937130 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:10.953377 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:10.971522 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:11.073413 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:11.162261 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:11.181835 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:11.272917 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:11.290083 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:11.326339 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:11.349637 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:11.359192 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:11.378567 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:11.387616 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:11.394564 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:11.405567 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:11.411623 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:11.451052 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:11.464769 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:11.481479 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:11.487787 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:11.581434 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:11.589386 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:11.606065 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:11.630138 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:11.668586 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:11.687166 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:11.707156 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:11.858567 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:11.866418 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:11.872958 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:11.885782 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:11.891822 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:11.908820 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:11.930508 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:11.932044 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:11.941529 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:11.943688 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:12.011307 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:12.021876 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:12.072397 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:12.092141 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:12.213856 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:12.243147 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:14:12.272184 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:12.321036 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:12.337682 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:12.389140 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:12.502717 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:12.512082 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:12.520608 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:12.523728 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:12.533643 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:12.553408 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:12.561655 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:12.614811 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:12.623236 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:12.631105 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:12.645594 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:12.885872 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:12.969874 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:12.996399 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:12.997377 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:13.026708 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:13.052278 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:13.053705 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:13.067824 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:13.092687 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:13.096756 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:13.098027 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.127267 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.141193 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:13.175571 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.185223 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.224629 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.245965 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:13.262650 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.268479 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:13.362273 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.419991 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:13.477901 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:13.499264 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:13.529294 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:13.536643 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:13.543617 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:13.555898 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:13.582393 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.587077 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:13.588189 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:13.594420 21034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.131:38349, user_credentials={real_user=slave}} blocked reactor thread for 57793us
W20240501 20:14:13.609282 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:14:13.619784 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:14:13.622365 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:13.623381 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:13.642654 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
I20240501 20:14:13.673134 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:13.684571 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:13.723618 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:13.752921 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:13.788187 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.832943 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:13.861323 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.890404 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.921301 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:13.996865 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:14.003862 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:14.025897 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:14.028154 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:14.084676 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 20:14:14.158380 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:14.172530 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:14.185407 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:14.207973 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:14.212450 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:14.233029 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:14.311473 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:14.347786 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:14.351431 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:14.355949 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:14.360165 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:14.366591 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:14.420876 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:14.459203 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:14.470873 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:14.486753 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:14.536751 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:14.540123 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:14.635280 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:14:14.672731 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:14.771205 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:14:14.772586 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:14.812503 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:14.821174 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:14.828909 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:14.832573 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:14.847659 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:14:14.853832 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:14.856984 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:14.873876 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:14.877969 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:14.920490 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:14.922853 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:14.953411 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:14.968628 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.017689 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:15.046468 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:15.122545 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:15.327060 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:15.327823 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:15.341409 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:15.345454 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:15.349936 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:15.350337 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:15.388660 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:15.396625 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:15.466223 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.474366 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.495090 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.531319 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.532362 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:15.678664 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.776556 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.789839 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:15.849874 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:15.869158 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:15.878579 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.893153 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.906993 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.918076 23910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 65609.7us
W20240501 20:14:15.919359 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.923858 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:15.950549 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:16.002578 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:16.016321 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:16.033397 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:16.072093 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:16.107632 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:16.151168 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:16.158730 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:16.176538 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:16.181435 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:16.220428 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:16.278941 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:16.326519 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:16.362520 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:16.404280 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:16.453305 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:16.453701 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:16.519987 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:16.561064 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:16.603207 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:16.615784 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:16.616451 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:16.632258 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:16.637168 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:16.637914 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:16.664691 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:16.664877 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:16.730386 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:16.807374 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:16.814282 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:16.823689 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:16.831068 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:16.844581 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:16.873970 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:14:16.879002 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:16.886565 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:16.898263 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:16.920161 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:16.947337 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:17.030117 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:17.288333 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:17.361390 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:17.371271 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:17.382962 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:17.384310 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:17.396751 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:17.399734 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:17.400475 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:17.420598 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:17.423772 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:17.442672 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:17.464679 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:17.473927 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:17.493095 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:17.533519 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:17.544389 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:17.567909 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:17.796543 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:17.886077 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:17.971740 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:18.017589 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:18.037423 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:18.039762 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:18.047436 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.051110 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:18.054625 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:18.070624 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:18.097695 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:18.112800 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.113653 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.133086 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:18.145185 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:18.145120 23647 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 104154us
W20240501 20:14:18.192612 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.201917 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:14:18.212977 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:18.290753 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.421370 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.423728 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.431330 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.451630 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.483994 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:18.569171 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.580637 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:18.581689 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:18.673660 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
I20240501 20:14:18.691711 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:18.715911 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 44629us
I20240501 20:14:18.717507 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:18.717921 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
I20240501 20:14:18.724499 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:18.724756 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:18.738691 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:18.740238 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:14:18.778072 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:18.780114 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:18.801429 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:18.855841 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:18.885852 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:14:18.945647 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:18.969241 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.988049 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:18.993506 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:19.002185 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:19.018599 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:19.094763 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:19.095220 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240501 20:14:19.096470 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:19.099591 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:19.175300 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:19.177815 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:19.199492 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:19.200910 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:19.204330 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:19.307029 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:19.324877 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:19.332173 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:19.367897 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:19.372879 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:14:19.378656 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:19.416644 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:19.437837 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:19.441206 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:19.454284 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:19.457010 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:14:19.486341 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:19.520184 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:19.568536 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:19.576480 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:19.607829 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:19.700896 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:19.723090 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:19.777376 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:19.875286 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:19.906733 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:19.910507 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:19.916898 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:19.952584 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:14:19.954742 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:19.968264 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:19.991062 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:20.026362 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.063779 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:14:20.067404 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:20.076783 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:20.267792 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:20.288388 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.306677 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.454192 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.456807 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.505677 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:14:20.518380 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:20.526038 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:14:20.531029 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:20.536327 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:20.549553 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:20.619405 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:20.692550 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:20.692960 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:20.715219 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:20.717213 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:20.731019 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:20.743710 21291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 52970us
W20240501 20:14:20.770123 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.786943 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.787004 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.795528 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.865953 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:20.874734 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.876263 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.901000 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.949707 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:20.949859 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:21.056799 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:21.101655 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:21.102881 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:21.124086 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:21.160549 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:21.189173 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:21.209317 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:21.251461 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:21.259099 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:21.361771 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:21.399559 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:21.401620 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:21.427713 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:21.436188 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:21.439268 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:21.440742 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:21.449687 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:21.551362 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:21.669895 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:21.678014 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:21.700328 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:21.729629 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:21.825332 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:21.834443 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:21.849812 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:21.852775 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:21.865051 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:21.881845 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:14:21.898826 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:21.900903 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:21.936990 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:14:21.954881 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:21.968137 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:14:22.047981 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:22.093297 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:22.094440 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:22.098506 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:22.133844 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:22.143062 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:22.164535 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:22.186372 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:22.193666 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:22.282084 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:22.375715 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:14:22.385275 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:22.388784 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:22.416980 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:22.421423 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:22.438280 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:22.466471 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:22.490741 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:22.498080 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:22.530378 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:22.668279 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:22.764485 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:22.817051 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:22.866420 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:22.885286 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:22.887624 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:22.896874 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:22.913034 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:22.922010 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:22.930845 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:22.936739 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:22.949896 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:22.969157 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:23.033989 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.162386 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.197516 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:23.199087 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.210898 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.226336 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.243114 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:23.256616 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.272660 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.315661 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:23.368680 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:23.370012 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.403450 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.545842 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.610169 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:23.685052 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:14:23.703389 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:23.705265 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:23.717689 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:23.719779 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:14:23.722605 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:23.730835 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:23.780866 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:23.781366 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:23.787632 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:23.814836 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:14:23.825512 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:23.837064 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:23.865892 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:23.885267 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.925877 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:23.984928 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:23.985431 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:24.105675 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:24.124694 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240501 20:14:24.160009 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:24.160466 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:24.185781 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:14:24.187099 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:24.206441 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:24.207188 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:24.230042 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:24.249210 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:24.310098 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:24.316533 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:24.337790 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:24.338127 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:24.338650 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:24.371125 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:24.384984 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:24.416656 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:24.475716 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:14:24.477876 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:24.535209 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:24.538074 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:14:24.572952 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:24.631902 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:24.633286 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:24.679104 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:24.701017 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:24.713575 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:24.789193 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:24.812808 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:24.824110 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:24.827159 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:24.830269 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:24.853704 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:24.869167 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:24.872956 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:24.895831 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:24.923193 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:24.950114 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:24.963011 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:24.968740 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:25.116402 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:25.186189 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:25.216634 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:25.225014 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:25.264719 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:25.431941 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:25.526015 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:25.565675 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:25.568078 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:25.572538 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:25.578437 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:25.591054 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:25.595103 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:25.615535 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:25.617278 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:25.650585 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:25.658957 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:25.669570 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:25.705703 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:25.715215 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:25.729024 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:25.737819 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:25.840195 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:25.880010 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:25.921384 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:25.975488 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:26.035091 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:26.057137 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:26.058068 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:26.084887 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:26.114675 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:26.118500 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:26.121883 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.174659 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:26.194512 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:26.264766 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:26.267249 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:26.366302 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.383225 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.476137 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:26.487882 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.512194 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:26.528893 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.586077 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.605751 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:14:26.608479 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:26.635741 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:26.648510 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:26.665747 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:26.665750 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:26.678076 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:26.698967 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:26.717835 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:26.726655 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.733295 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:26.736814 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:26.775532 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.806533 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.903724 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:26.951258 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:26.970794 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:27.023411 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:27.074333 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:14:27.078574 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:27.119269 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:27.141955 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:27.142885 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:27.262962 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:27.285450 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:27.304605 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:27.333384 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:27.339217 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:27.350042 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:27.352195 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:27.381168 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:27.418993 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:27.429550 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:27.474541 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:27.484213 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:27.505899 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:27.506690 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:27.536841 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:27.543002 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:27.617892 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:27.619064 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:27.872051 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:27.877099 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:27.892274 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:27.893327 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:27.899724 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:27.902295 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:27.920190 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:27.958081 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:27.959118 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.004223 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:28.015641 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.062934 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:28.107353 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.129055 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.184460 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.197885 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.241307 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.242606 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.358021 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:28.366254 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:28.367254 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:28.376070 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:28.416924 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:28.442911 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:28.449333 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:28.462651 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:28.504098 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:28.513958 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:28.539974 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:28.573021 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:28.579851 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:28.619889 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.675983 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:28.700076 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240501 20:14:28.727216 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:14:28.745213 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:28.754437 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.863821 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.985595 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:28.997713 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:29.066830 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:29.090054 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:14:29.090137 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:29.095825 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:29.125838 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:29.156067 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:29.165454 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:29.201602 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:29.207613 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 20:14:29.232630 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:29.275418 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:29.293447 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:29.304744 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:29.308579 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:14:29.313305 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:29.315611 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:29.317430 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:29.324199 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:29.328341 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:29.340027 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:29.347441 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:29.418458 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:29.603466 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:29.624269 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:29.625597 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:29.719586 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:29.767192 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:29.768762 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:29.780112 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:29.780361 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:29.780333 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:29.806849 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:29.817616 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:29.818033 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:29.821856 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:29.830965 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:29.832556 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:29.843123 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:14:29.851987 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:29.854019 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:29.871999 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:29.880353 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:29.922780 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:29.957962 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:30.141058 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:30.230851 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:30.231477 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:30.287633 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:30.324065 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:30.338260 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:30.359232 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:30.397972 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:30.425016 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:30.426437 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:30.444900 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:30.450626 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:30.486210 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:30.499437 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:30.509414 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:30.547407 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:30.711261 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:30.724222 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:30.728380 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:30.758687 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:30.969434 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:30.987488 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:30.991673 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:30.992141 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:31.015942 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:31.027297 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:31.036684 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:31.091926 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:31.094012 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:31.094982 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:31.095630 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:31.119439 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:31.152139 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 44128.6us
W20240501 20:14:31.175725 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:31.180469 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:31.183902 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:31.213192 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:31.229763 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:31.281241 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:31.393988 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:31.408288 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:31.506913 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:31.549472 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:31.573029 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:31.596765 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:31.610455 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:31.631079 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:31.631649 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:31.644208 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:31.645516 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:31.668411 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:31.683734 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:31.712167 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:31.719482 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:31.727463 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:31.843915 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:31.844406 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:31.865090 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:31.902132 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:31.981921 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:31.983867 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:32.034642 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:32.074990 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:32.090276 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:32.090422 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:14:32.097290 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:32.131966 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:32.147620 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:32.215749 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:32.232890 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:32.233901 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:32.240926 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:32.243158 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:32.257221 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:32.269084 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:32.298976 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:32.394099 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:32.427919 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:32.432538 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:32.458034 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:32.471329 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:32.476547 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:14:32.544456 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:32.605652 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:32.789371 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:32.823405 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:32.837332 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:32.855417 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:32.949014 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:32.952435 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:32.961673 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:32.970968 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:32.981186 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:33.004254 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:33.026558 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:33.084944 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:33.091526 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:33.145388 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:33.152607 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:33.286114 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:33.300168 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:33.355367 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:33.399744 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:33.441785 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:33.503036 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:14:33.531793 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:33.538398 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:33.540052 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:33.540980 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:33.564841 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:33.565953 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:33.598479 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:33.611728 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:33.625876 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:33.632877 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:33.644409 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:33.686259 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:33.717813 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:33.745291 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 20:14:33.746238 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:33.798764 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:14:33.866582 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:33.948484 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:33.954063 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:33.983670 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:34.004567 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:34.014209 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:34.035027 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:34.090680 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:34.102224 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:34.115694 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:34.120158 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:34.128324 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:34.188292 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:34.218981 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:34.227986 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:34.230960 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:34.231416 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:34.247642 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:34.343712 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 20:14:34.365387 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:34.373894 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:14:34.374756 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:34.409302 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:34.411041 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:14:34.468319 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:34.514294 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:34.568905 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:34.579000 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:34.584517 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:34.628420 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:34.639178 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:34.654042 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:34.673286 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:34.749402 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:34.753075 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:34.758237 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:34.863351 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:34.903271 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:34.906742 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:14:34.915114 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:34.922049 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:34.956007 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:34.979197 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:35.014026 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:35.100477 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:35.101269 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:35.104948 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:35.115576 20902 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 50214.6us
W20240501 20:14:35.194223 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:35.235572 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:35.235746 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:35.250217 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:35.365347 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:14:35.441308 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:35.463811 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:35.466503 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:35.507977 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:14:35.508011 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:35.521050 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:35.535897 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:35.572052 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:35.594786 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:14:35.595196 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:35.608451 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:35.632082 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:35.755031 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:35.755282 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:35.820981 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:35.914734 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:35.915143 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:36.002411 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:14:36.013198 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:36.014889 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:36.015973 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:14:36.025646 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:36.057615 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:36.069729 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:36.085197 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:14:36.090088 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:36.122493 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:36.123713 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.123836 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:36.209717 23779 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.129:41259, user_credentials={real_user=slave}} blocked reactor thread for 88306.6us
W20240501 20:14:36.211500 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.213744 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.280562 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.366394 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.368144 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:36.371106 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:36.422811 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.498898 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:36.545246 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:36.585088 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:36.586529 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.675868 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:36.722668 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.758313 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.762620 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.768620 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:36.828377 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:14:36.893977 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:36.896855 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:36.908087 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:14:36.911705 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:36.914889 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:36.916103 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:14:36.917565 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:36.927040 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:14:36.944128 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:36.956578 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:36.960855 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:36.975335 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:36.980165 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:36.997421 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:37.062141 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:37.072752 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:37.081439 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:37.192528 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:37.216866 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:37.408469 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:37.437129 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:37.488132 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:14:37.509600 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:37.528172 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:37.531791 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:37.553088 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:37.553784 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:37.562088 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:37.582521 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:14:37.594034 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:37.612365 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:37.670281 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:37.768342 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:37.785156 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:37.966766 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:37.973659 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:37.974246 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:37.991492 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:37.999301 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:38.003191 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:38.003800 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:38.004966 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:14:38.012657 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:38.026320 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:38.040791 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:38.097537 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.124321 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.189275 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.265883 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:38.313810 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.324827 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.372910 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.418339 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:38.500819 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.609678 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.632411 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:14:38.747880 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:38.757699 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:38.778499 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.782604 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240501 20:14:38.784415 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:38.786545 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:38.787057 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:38.787722 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:38.802837 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:38.839754 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:38.845095 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:38.857772 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:38.866343 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 122 similar messages]
I20240501 20:14:38.876148 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:38.942289 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:38.946384 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:38.946403 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:39.016840 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.060503 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.071785 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.078620 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.183020 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.215346 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:39.299934 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:39.309643 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:39.335595 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:14:39.338500 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:39.356038 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:39.367517 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:39.369200 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:39.396451 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:39.396908 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:39.407526 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:39.408226 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:39.409096 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240501 20:14:39.415510 24043 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 67822.9us
W20240501 20:14:39.420368 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:39.420480 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:39.424804 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:39.449719 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.456593 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:39.459010 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:39.459895 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.490404 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.554553 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.601706 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.707990 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:39.709084 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.810719 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:39.813151 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:39.888470 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:39.921442 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:39.923132 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:39.923589 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:39.937222 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:39.962143 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:39.965050 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:39.977954 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:39.979218 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:40.107551 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:40.108312 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:14:40.114919 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:40.217877 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:40.314266 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:40.370024 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:40.383853 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:40.389252 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:40.404013 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:40.469545 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:40.596227 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:40.673192 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:40.675767 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:40.691967 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:40.720237 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:40.720767 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:40.754588 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:40.802835 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:40.807425 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:40.809458 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:40.831522 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:40.892980 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:40.902081 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:40.929467 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.000190 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.048748 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.074049 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.110766 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:41.227404 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:41.281915 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:41.290557 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:41.292708 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:41.301704 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:41.303658 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:41.354024 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:14:41.381330 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:41.387400 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:41.388147 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:41.414470 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:41.421445 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:41.444322 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:41.452723 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:41.462682 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.479527 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:41.579643 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:41.674239 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.719290 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.751760 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.765153 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.777494 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:41.779582 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:14:41.791394 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:41.819224 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:14:41.839249 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:41.841128 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:41.843175 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:41.880100 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:41.882292 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:41.889644 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:41.894976 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:41.901340 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:41.901465 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:41.929296 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:41.956475 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:41.971675 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:41.972157 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:41.999100 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:42.014365 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:42.024808 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:42.030642 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:42.096705 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:42.186854 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:42.239107 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:42.336256 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:42.483737 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:14:42.561759 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:42.562815 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:42.582697 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:42.590677 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:42.614995 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:42.648205 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:42.655602 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:14:42.660030 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:42.693734 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:14:42.700660 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:42.704504 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:42.741360 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:42.758814 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:42.765162 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:42.931085 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:43.024384 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.101570 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.145498 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.170044 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:14:43.175150 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:43.187733 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:43.193971 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.201246 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:14:43.263790 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:43.270890 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:43.297093 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:43.298943 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:43.305447 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:43.346186 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:43.441993 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.447626 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:43.484124 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.515616 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:43.540195 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.594643 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.667531 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:14:43.669526 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:43.683436 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:43.694375 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:43.705794 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:43.715158 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:14:43.737416 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:43.773993 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:43.804903 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:14:43.813328 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:43.825865 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
I20240501 20:14:43.836796 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:43.926697 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:14:43.945582 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.948812 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:43.958971 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:43.969357 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.093556 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:44.131464 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:44.158752 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.254407 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:44.289324 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:44.295825 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.322310 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.385845 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:44.408017 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:44.413975 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 20:14:44.427058 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.453702 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:44.462342 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.500731 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.515228 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:44.517915 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.522883 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:44.523288 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:14:44.544070 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:44.546671 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:14:44.548163 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:14:44.554461 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:44.559958 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:44.570554 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:44.587283 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:44.597182 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:14:44.611582 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:44.619388 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:44.625327 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.689568 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:44.726684 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:44.797856 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:44.869834 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:44.877072 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:14:44.915398 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:44.997223 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:45.004842 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:45.008966 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:45.012450 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:45.014202 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:45.014480 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:14:45.041584 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:45.063184 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:45.103941 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:45.177253 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.247983 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.311489 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.332893 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.366997 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.438813 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.481160 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:45.516968 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:14:45.543828 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:45.544198 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:45.544934 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:45.555385 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:45.560683 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:45.569191 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:45.589649 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:14:45.591143 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:45.598989 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:45.624289 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:45.695362 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:45.835074 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.884341 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.967273 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:45.967734 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.976357 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:45.989809 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:46.013307 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:46.061025 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:46.111945 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:46.116099 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:46.319689 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:46.348158 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:14:46.355458 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:46.381093 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:46.399624 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:46.412211 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:46.428236 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:46.437265 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:46.438176 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:46.465788 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:14:46.467068 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:46.488706 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:46.513226 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:46.548517 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:46.619148 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:46.632479 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:46.660795 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:46.669494 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:46.718595 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:46.790246 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:46.813593 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:46.831439 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:14:46.855434 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:46.897059 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:46.907277 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:46.908732 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:46.912741 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:46.913619 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:46.918295 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:14:46.923132 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:46.982044 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:47.034293 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:47.060612 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:47.075872 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:47.105919 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:47.122246 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:47.125447 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:47.131386 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:47.132038 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:47.173030 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:47.237644 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:47.319512 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:47.360181 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:47.374711 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:47.428458 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:47.437460 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:14:47.444904 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:47.457347 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:47.483050 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:47.522269 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:47.529440 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:47.556787 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:47.589179 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:47.606094 23644 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.130:42271, user_credentials={real_user=slave}} blocked reactor thread for 50286us
W20240501 20:14:47.672791 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:47.732213 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:47.748620 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:47.803175 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:47.855513 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:47.890476 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:47.957123 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:47.985446 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.177114 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:48.234673 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:48.283774 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.291661 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.334291 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.354049 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.368032 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:48.377425 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:48.397599 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:14:48.401798 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:48.407894 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:48.429841 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:48.457083 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:48.487190 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:48.487661 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:48.570752 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:48.573228 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.672438 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.673321 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.680280 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:48.732172 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.775499 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.849434 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:48.866746 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
I20240501 20:14:48.902660 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:48.937460 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:14:48.938930 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:48.949875 20902 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 46735.5us
I20240501 20:14:48.976227 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:48.990432 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:48.992246 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:48.993770 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:48.998611 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:49.052927 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:49.062343 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:49.077967 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:49.103387 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:49.106814 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:49.139094 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:49.162921 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:49.163084 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:49.192366 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:49.229727 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:49.378799 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:49.386597 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:49.435010 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240501 20:14:49.473865 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:49.487726 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:49.503800 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:49.520828 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:49.541491 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:49.560453 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:14:49.573839 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:49.590597 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:49.599792 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:49.600410 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:49.613786 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:49.623044 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:49.639688 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:49.662081 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:49.666069 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:49.696648 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:49.788055 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:49.800311 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:49.808697 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:14:49.811064 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:49.812047 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:49.863564 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:49.918826 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:50.038520 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:50.057230 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.067921 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.177206 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.227097 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:50.248232 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.373700 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.377283 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:50.385866 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:50.414830 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:50.439388 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:50.442144 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:50.445688 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:50.471375 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:50.497220 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:50.502430 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:50.562541 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:50.604344 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.641963 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.642947 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.673168 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:14:50.838493 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.847208 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:50.878618 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:50.878772 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:50.930400 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:50.979043 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:51.004804 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:51.017554 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:51.030841 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:51.031061 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:51.057518 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:51.073837 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:51.075186 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:51.075626 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:51.086115 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:51.117398 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:51.169667 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:51.301488 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:51.327667 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:51.412364 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:51.416217 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:51.458312 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:51.490669 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:51.492643 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:51.565106 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:14:51.572587 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:51.572734 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:51.638768 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:51.655915 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:51.682518 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:51.692238 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:51.697839 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:51.724319 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:14:51.728410 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:51.757769 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:51.770718 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:51.790056 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:51.848932 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:51.865841 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:51.873427 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:51.880236 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:14:52.050585 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:52.082963 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:52.104091 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:52.140069 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:52.188127 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:52.195103 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:52.251475 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:52.270563 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:52.311731 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:14:52.389061 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:52.391027 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:14:52.424643 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:52.432610 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:52.436542 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:52.454850 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:52.476540 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:52.483687 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:14:52.493111 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:52.505954 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:52.525923 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:52.610419 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:52.678156 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:52.733454 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:52.768182 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:52.785634 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:52.818297 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:52.845527 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:14:52.896030 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:52.914357 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:52.919579 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:52.941927 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:52.962013 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:53.036347 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:53.059862 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:53.064219 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:53.098389 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:53.120472 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:53.124398 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:53.126884 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:53.209277 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:53.258850 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:53.275240 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:53.285959 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:53.346576 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:53.447275 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:53.453023 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:14:53.473178 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:53.500921 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:53.509860 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:14:53.519521 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:53.538218 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:53.539811 23647 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 47946.2us
W20240501 20:14:53.544545 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:53.559783 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:53.568951 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:53.572052 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:53.605717 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:53.605775 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:53.641803 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:53.825474 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:53.958344 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 20:14:53.974262 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:14:54.037369 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:54.054138 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:54.068714 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:54.076242 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:54.077639 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:54.198390 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:54.233177 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:54.252758 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:54.271595 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:54.282263 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:54.284211 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:54.308887 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:54.326871 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:54.334235 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:54.335357 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:54.336117 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:14:54.348731 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:54.370955 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:54.402688 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:14:54.416384 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:54.460637 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:54.516295 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:54.543098 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 20:14:54.544112 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:54.579676 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:54.597404 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:14:54.600831 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:54.622153 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:54.772658 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:14:54.796892 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:54.801518 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:54.803508 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:54.812070 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:14:54.816345 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:54.839818 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:54.855829 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:54.865060 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:54.870817 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:54.884725 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:14:54.891009 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:54.893646 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:54.967604 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:55.044483 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:55.074271 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:55.117712 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:55.119098 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:55.129577 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:55.177541 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:55.305043 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:55.338867 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:55.384454 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:55.388693 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:55.397004 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:55.397171 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:55.413599 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:55.419804 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:55.434300 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:55.440299 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:55.471971 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:14:55.500243 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:55.500911 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:55.567277 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:55.601727 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:55.653105 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:55.691377 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:55.716414 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:55.729017 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:55.755347 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:55.810341 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:55.877727 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:55.892609 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:55.970319 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:56.023241 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:56.030056 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:14:56.084758 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:56.118028 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:56.125543 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:56.126228 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:56.127796 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:56.143748 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:56.146207 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:56.186398 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:56.206691 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:56.263643 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:56.346037 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:56.395062 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:56.575320 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:56.642848 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:56.642933 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:56.643241 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:56.681198 21293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.133:37155, user_credentials={real_user=slave}} blocked reactor thread for 46976.4us
I20240501 20:14:56.687484 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:56.702623 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:56.716089 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:56.780725 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:56.788492 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:56.791499 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:56.797292 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:56.806663 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:56.819051 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:56.829519 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:56.849270 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:56.856398 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:56.859987 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:56.937459 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:56.941291 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:57.000468 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:57.018980 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:57.037829 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:57.128083 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:57.163066 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:57.193058 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:57.229104 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:57.318678 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:57.319816 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:57.322301 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:57.331763 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:14:57.346153 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:14:57.355571 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:57.357985 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:57.360204 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:57.399390 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:57.429327 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:57.430859 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:57.461009 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:57.471940 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:57.492362 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:14:57.552608 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:57.644256 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:57.666400 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:57.698503 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:57.810899 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:57.903375 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:57.936728 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:58.020064 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:58.031525 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:14:58.032629 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:58.048357 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:58.068960 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:58.069360 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:14:58.111517 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:58.115648 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:14:58.119819 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:58.175194 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:58.205549 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:58.233006 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:14:58.245818 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:58.275599 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:58.335834 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:58.361907 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:58.381176 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:58.409056 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:58.459829 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:58.579319 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:14:58.595630 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:58.606447 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:58.615911 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:14:58.616607 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:58.635697 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:14:58.641901 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:58.644698 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:58.704806 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:14:58.729261 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:14:58.741689 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:58.783246 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:58.823786 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:58.861582 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:58.925168 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:59.041795 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 20:14:59.042711 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:59.047073 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:59.047681 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:59.082075 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:14:59.093647 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:59.127795 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:14:59.171909 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:59.172178 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:59.206430 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240501 20:14:59.206657 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:59.209589 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:14:59.213202 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:14:59.220171 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:59.235023 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:14:59.275671 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:59.279760 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:14:59.310294 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:14:59.330138 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:59.406957 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:59.443272 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:59.457883 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:59.487216 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:59.502274 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:14:59.540493 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:14:59.592482 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:59.615710 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 20:14:59.621744 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:14:59.632731 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:59.702179 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:59.733920 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:14:59.809746 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:14:59.842763 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:59.854903 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:59.867513 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:14:59.901814 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:14:59.902410 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:59.937726 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:14:59.947466 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:14:59.952132 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:14:59.954038 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:14:59.974715 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:14:59.987799 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:59.994617 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:14:59.995352 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:15:00.008545 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:00.026702 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:00.072336 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:15:00.145881 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:00.342581 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:00.380707 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:15:00.404942 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:00.420078 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:00.444895 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:00.484391 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:00.497710 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:00.508350 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:00.551800 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:00.558482 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:00.561703 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:00.564471 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:15:00.600847 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:15:00.640914 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:00.690413 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:00.755241 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:00.797663 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:00.833374 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:00.875505 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:15:00.877458 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:00.895689 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:00.911087 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:00.975054 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:00.975420 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:00.982944 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:00.989940 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:00.992830 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:01.002759 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:01.016913 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:15:01.021828 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:01.027240 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:01.064596 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:01.108583 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:01.172609 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:01.203635 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:01.254098 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:01.285271 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:01.291769 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:01.365236 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:01.491000 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:01.546887 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:01.558604 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:01.627775 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 62843us
W20240501 20:15:01.690063 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:15:01.700475 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:15:01.708865 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:01.710947 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:15:01.720589 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:01.724439 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:01.745138 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:01.775766 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:01.783213 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:01.785737 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:01.787925 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:01.824784 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:01.858029 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:01.915702 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:01.965237 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:02.073397 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:02.091212 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:02.113802 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:02.142400 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:02.167098 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:02.183665 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:15:02.193864 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:02.240339 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:02.241573 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:02.244575 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:02.269157 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:02.306514 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:02.307475 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:02.351022 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:02.372371 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:02.384387 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:02.394213 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:15:02.407673 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:02.436748 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:02.437876 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:02.450202 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:02.484870 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:02.497682 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:02.657409 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:02.722865 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:02.831663 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:02.851964 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:02.870714 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:02.890739 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:02.895766 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:02.905452 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:02.975009 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:02.976047 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:03.008839 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:15:03.009654 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:03.015130 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:03.026160 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:03.064663 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:03.069603 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:03.075371 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:03.101023 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:03.199092 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:03.206269 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:03.359611 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:03.422914 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:03.449579 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:03.477887 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:03.530611 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:03.580739 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:03.582267 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:03.598981 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:03.642161 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:03.648046 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:03.657155 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:03.660835 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:03.706902 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:03.734887 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:03.766968 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:03.779994 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:03.784869 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:03.784963 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:03.890725 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:03.907711 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:03.919844 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.101593 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.106696 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:15:04.131135 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.131276 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:04.135082 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:04.160028 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:04.165666 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:04.191116 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:04.199198 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:04.216241 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
I20240501 20:15:04.220397 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:04.251935 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:04.252508 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:04.283699 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:04.297348 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:04.360071 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:04.404301 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.427690 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:04.448176 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.464164 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:04.481187 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.612099 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.627831 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:15:04.629959 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.659011 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:04.674072 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:04.676216 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:15:04.724484 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.747792 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:04.785072 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240501 20:15:04.790079 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:04.797356 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:04.799125 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:04.815943 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:04.819773 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:04.830675 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:04.838410 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:04.847379 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:04.890697 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:04.902561 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:04.917536 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:04.964334 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:04.967482 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:15:04.972211 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:04.979777 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:05.216161 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:05.234560 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:05.335655 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:05.347710 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:05.400761 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:05.508823 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:05.536834 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:05.536818 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:05.554764 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:05.559350 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:05.584316 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:05.592264 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:05.594312 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:05.648756 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:05.669836 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:05.677812 20900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.135:41019, user_credentials={real_user=slave}} blocked reactor thread for 70628.4us
W20240501 20:15:05.691079 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:05.706707 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:05.777463 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:05.846791 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:05.862383 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:05.862965 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:05.923161 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:05.939723 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:15:05.942193 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:05.976783 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:15:06.010213 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:06.024806 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:06.025566 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:06.026140 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:06.029256 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:06.030447 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:06.043072 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:06.076061 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.093502 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:06.110522 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:06.213192 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.230510 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:06.243103 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.293850 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.300478 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:06.369977 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:06.491060 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.524621 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:15:06.526870 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:06.535656 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.555316 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.590075 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:15:06.591961 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:06.595906 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:06.596364 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:06.607841 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:06.626806 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:06.684736 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:06.692500 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:06.730340 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:06.820673 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.829164 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.912086 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:06.935849 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:06.971948 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:07.028462 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:07.029803 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:07.046599 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:07.052866 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:07.070758 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:07.139587 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:07.234207 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:15:07.300348 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:07.318250 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:07.337543 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:15:07.371129 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:07.408651 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:07.416023 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:07.476738 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:07.479512 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:07.479529 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:07.504483 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:15:07.507892 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:07.517563 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:07.554785 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:07.562592 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:07.564538 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:07.625036 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:07.703823 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:07.838505 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:07.923729 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:07.924245 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:07.936005 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:15:07.939795 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:07.941318 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:07.943598 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:07.953233 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:08.002770 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:08.025965 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:08.029215 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.144033 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.152006 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:08.234091 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:08.240051 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.247256 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.275642 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.363715 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:08.372756 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.395267 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.416347 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.490398 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:15:08.498245 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:08.499282 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:08.523777 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:08.536578 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:08.558873 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:08.611824 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:08.619658 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:08.664690 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:08.670377 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:08.677228 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.698657 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.771000 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.785091 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:08.829166 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:08.874989 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:08.951109 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:08.968235 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:08.980958 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.004038 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:09.110913 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:09.135826 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.145977 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240501 20:15:09.173434 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.212149 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.221416 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.242620 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:15:09.243680 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.260061 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.271656 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:09.289784 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:09.298584 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:09.303580 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:09.309816 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:09.311949 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:09.331027 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:09.375218 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:15:09.385470 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:09.411227 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.481137 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:09.505662 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.517642 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.641710 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:15:09.687964 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:09.695012 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.727272 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:09.765750 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:09.766074 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:09.766698 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:09.771085 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:09.806375 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:09.807216 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:09.809568 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:09.811821 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:09.817307 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240501 20:15:09.822554 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:09.866448 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:09.912240 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:09.946563 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:15:10.018988 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:10.048869 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:15:10.066385 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:10.103991 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:10.145987 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:10.238682 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:10.331880 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:10.334980 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:10.335189 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:10.337399 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:15:10.339191 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:10.340245 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:10.349954 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:10.351145 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:10.445762 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:10.478742 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:15:10.488070 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:10.497279 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:10.608672 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:10.714098 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:10.719182 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:10.723029 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:10.731603 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:10.882525 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:10.889672 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:10.906349 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:10.965332 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:15:11.023502 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.050590 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:11.059316 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:11.077793 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.084256 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:11.086220 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:11.102483 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:11.114432 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:11.135279 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:11.147400 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:11.168390 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:11.273598 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:11.275648 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:11.288333 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:11.326277 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:11.339603 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:11.395938 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.420338 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.423661 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.475252 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.525565 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.534200 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:11.621404 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:11.623572 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:11.694114 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:11.695389 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:11.704133 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:11.716351 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:11.721182 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:11.761286 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.771201 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:11.790567 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:11.805544 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:11.812064 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:11.828693 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:11.849550 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:11.849840 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.913619 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:11.945868 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:11.967150 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:12.092557 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:12.114917 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:12.131531 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:12.155216 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:12.157182 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:12.173143 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:12.181978 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:12.194437 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:12.215322 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:12.220299 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:12.246501 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:12.294050 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:12.307833 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:12.322952 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:12.378737 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:15:12.422540 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:12.652685 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:12.675865 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:12.691453 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:12.730508 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:12.848027 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:12.852216 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:12.871449 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:12.887367 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:12.887822 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:12.901486 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:12.912938 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:12.918304 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:12.938802 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:12.968408 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:13.017275 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:13.098599 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.101495 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.118554 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.150363 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.232671 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:13.284242 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:13.308861 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:13.382001 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:13.389308 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:13.405110 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:13.408761 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:13.424882 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.433524 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:13.463078 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:13.496088 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:13.500252 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:13.505748 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.507702 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:13.540750 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:13.581984 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:13.645694 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.661505 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.705657 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.804366 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:13.806870 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:13.870697 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:13.894766 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:13.912532 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:13.921571 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:13.928791 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:13.929847 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:13.966684 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:15:13.968426 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:13.977269 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:13.994771 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:13.996120 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:13.999646 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:14.033438 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:14.062750 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:14.098760 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:14.144934 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:14.199620 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:15:14.200440 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:14.232616 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:14.316537 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 20:15:14.324311 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:14.403930 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:14.405867 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:14.443485 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:14.463251 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:14.479405 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:14.503199 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:14.560586 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:15:14.642537 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:15:14.673130 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:14.677624 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:14.697396 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:14.697970 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:14.718842 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:14.748265 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:14.752285 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:14.757375 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:14.762944 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:14.809234 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:14.833489 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:14.841406 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:14.866719 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:14.893684 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 20:15:14.894598 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:14.968029 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:15.018699 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:15.168426 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:15.170539 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:15.191851 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:15.207234 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:15.226119 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:15:15.234287 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:15.238536 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:15.265547 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:15.267589 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:15.290611 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:15.321233 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:15.341193 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:15.354498 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:15.366607 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:15.452212 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:15.522979 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:15.549293 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:15.639400 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:15.711735 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:15.713447 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:15.773486 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:15.778213 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:15.798410 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:15.811277 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:15.823288 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:15.827069 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:15.828487 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:15.899516 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:15.902071 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:15.902264 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:15.920125 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:15.939596 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:15.945837 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.010411 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.022486 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:16.062165 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.086725 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:15:16.105443 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.338017 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.350239 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:16.370008 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.414930 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:16.453150 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:16.465786 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:16.473697 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:16.476560 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:16.476819 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:16.484741 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:16.499346 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:16.503846 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:16.509529 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:16.524441 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.558317 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:15:16.562268 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.562400 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:16.567662 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:16.575796 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:16.590968 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.595911 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:16.643340 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:16.699195 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:16.835876 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:16.885747 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:16.952980 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:16.953656 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:16.955111 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:16.957334 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:16.979741 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:16.982335 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:16.994410 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:15:17.003147 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:17.003438 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:17.015131 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:17.051017 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:17.251739 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:17.372730 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:17.374758 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:17.387529 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:17.435853 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:17.444888 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:17.449513 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:17.481478 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:17.484305 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:17.498687 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:17.518746 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:17.530170 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:17.531006 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:17.562024 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:17.576440 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:17.621073 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:17.623292 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:17.657070 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:17.742523 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:17.791450 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:17.838230 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:17.845088 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:17.903218 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:17.935279 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:17.999913 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:18.002911 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:18.202579 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:18.206578 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:18.242025 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:18.247169 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:18.260242 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:18.265508 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:18.298921 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:18.309628 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:18.329573 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:18.356494 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:18.357147 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:18.372423 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:18.421838 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:18.426400 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:18.438239 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:18.469017 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:18.487859 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:18.590425 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:18.602432 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:18.787622 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:18.787981 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:18.790527 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:18.822101 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:18.837136 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:18.844461 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:15:18.847680 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:18.874411 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:18.911815 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:18.937314 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:18.942305 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:18.954675 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.029222 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.070389 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.088058 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:19.091936 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.135138 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:19.174914 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.218123 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:19.253279 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:15:19.254516 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.286689 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:19.317690 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:19.318379 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:19.341497 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:19.345868 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:19.355860 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
I20240501 20:15:19.360352 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:19.378520 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:19.397497 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.412220 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:19.413957 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:19.424206 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:19.448381 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:19.503979 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.593899 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.652711 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:15:19.725878 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:19.785034 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:15:19.789119 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.814479 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.890736 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:19.913934 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:19.915367 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:19.950940 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:19.951022 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240501 20:15:19.973418 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:19.986972 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:19.989116 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:20.013808 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:20.023607 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:20.031764 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:20.039400 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:20.080386 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:20.087795 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.088485 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:20.096436 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.114001 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:20.137387 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.146831 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.231930 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:15:20.337213 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:20.368542 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:20.379869 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.448050 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.522017 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:15:20.522169 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:20.536217 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:20.544149 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:20.552152 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:20.555276 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:20.561818 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:20.578305 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:20.625651 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:20.713486 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.772207 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.784315 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.825415 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.833052 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:20.862942 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:20.955935 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:20.989274 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:21.004748 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:21.006163 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:21.015940 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:21.019273 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:21.029218 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:21.038110 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:21.044361 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:21.057998 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:21.071617 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:21.078446 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:21.112275 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:21.292599 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:15:21.293582 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:21.322522 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:21.337354 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:21.409899 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:21.485479 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:21.608371 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:21.628439 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:21.647478 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:21.675827 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:21.725447 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:21.737505 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:21.798218 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:21.798698 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:21.803594 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:21.835083 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:21.835534 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:21.837131 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:21.850409 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:21.855799 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:21.909274 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:21.954290 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:21.966749 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:21.996157 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:22.053191 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:22.192268 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:22.227979 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:22.233310 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:22.235466 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:22.243584 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:22.256175 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:22.307219 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:22.316987 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:15:22.346272 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:22.365265 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:22.369942 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:22.377911 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:22.388711 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:22.399770 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:22.414306 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:22.456610 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:22.468493 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:22.470819 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:22.482764 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:22.529856 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:22.551208 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:22.551641 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:22.624616 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:22.734351 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:22.743089 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:22.758256 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:22.801283 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:22.840587 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:22.840734 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:22.842032 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:22.842662 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:22.848448 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:22.849398 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:22.890450 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:22.901753 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:22.906394 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:22.949429 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:23.130342 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:23.134708 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:23.166378 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:23.201431 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:23.285893 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:23.301519 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:23.338486 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:23.373498 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:15:23.471127 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:23.477828 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:23.524802 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:23.574738 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:23.579759 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:23.581825 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:23.597138 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:23.598526 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:23.623842 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:23.631915 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:23.654832 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:23.668483 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:23.703786 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:23.711042 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:23.770226 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:23.857237 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:23.898234 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:23.964020 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:24.007441 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:24.015830 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:24.031960 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:24.070212 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:24.112478 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:24.124200 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:24.198889 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:24.209308 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:24.241820 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:24.262189 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:24.275621 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:24.286454 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:15:24.288403 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:24.313493 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:24.314814 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:24.379737 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:24.396868 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:24.516563 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:15:24.557917 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:24.595194 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:24.648901 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:24.667418 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:24.702992 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:24.722802 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:24.733716 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:24.744630 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:24.747951 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:24.767078 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:24.767498 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:15:24.772661 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:24.795351 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:24.802848 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:24.802987 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:24.822908 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:15:24.842386 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:24.849211 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:24.867774 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:15:24.879714 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:24.884488 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:24.903124 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:25.005683 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:25.005720 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.021457 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240501 20:15:25.023301 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.023612 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:25.026165 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:25.092543 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.186703 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.249521 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:15:25.323693 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:25.411931 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:25.449123 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:25.454946 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:25.522576 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:25.530160 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:25.531054 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:25.548333 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:25.555567 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:25.605360 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.620282 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.655256 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:25.656123 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.704156 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.841579 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.880605 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:25.920329 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:25.942387 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:25.942518 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:26.029738 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:26.038091 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:26.043430 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:26.046811 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:26.047783 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:26.048825 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:26.060632 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:15:26.060791 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:26.088369 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:26.105669 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:26.139094 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:26.140868 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:26.146358 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:26.160223 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:26.285295 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:26.382890 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:15:26.411248 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:26.472294 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:26.549105 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:26.553296 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:26.556025 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:26.591117 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:26.601613 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:26.626209 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:26.626278 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:26.704341 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:26.709929 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:26.743824 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:26.790496 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:26.818029 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:26.829671 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:26.861805 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:26.868067 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:26.892166 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:26.948526 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:27.047966 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:27.077785 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:27.086426 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:27.093273 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:27.208374 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:27.208438 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:27.241202 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:27.260169 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:27.272852 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:27.351218 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:27.355480 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:27.363327 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:27.382542 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:27.390342 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:27.402272 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:27.414060 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:27.438511 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:27.439256 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:27.446715 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:15:27.450572 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:27.457177 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:27.459153 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:27.483515 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:27.524698 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:27.560673 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:27.591060 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:27.612483 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:27.706221 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:27.715063 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:27.762920 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:27.872778 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:27.894850 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:27.899487 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:27.928185 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:27.943943 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:27.948206 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:27.974225 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:28.072897 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:28.104630 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:28.179513 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:28.217981 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:28.268023 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:28.272198 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:28.345831 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:28.353636 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:28.413718 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:15:28.434610 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:28.444343 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:28.446483 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:28.457552 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:28.498148 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:28.517539 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:15:28.533944 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:28.541975 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:28.563097 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:28.569231 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:28.569363 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:28.573267 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:28.596746 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:15:28.596943 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:28.793372 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:28.814630 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:28.907191 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.035481 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.106563 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:29.150925 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:29.170295 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:15:29.192234 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:29.217212 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:29.232765 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:29.258266 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.296424 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:15:29.309893 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.318756 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:29.378969 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:29.399801 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:29.424425 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:29.432575 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:29.464777 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:15:29.467430 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:29.537688 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.554441 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:15:29.555356 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:29.566874 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:29.630985 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.693192 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.745352 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:29.758073 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.784690 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.792439 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:29.819437 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.836051 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:29.853675 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:29.856386 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:29.878185 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:29.888520 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:29.904474 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:29.907359 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:29.922097 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:29.955094 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:29.972779 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:29.995168 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:29.999413 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.029700 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.032424 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 20:15:30.050973 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:15:30.062259 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:30.062361 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:15:30.082800 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:30.188807 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:30.207504 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.241891 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.258054 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.387491 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:30.391863 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:30.404810 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:30.405797 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:30.430024 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:30.432754 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:15:30.438531 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:30.461134 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:30.483808 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:30.496059 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.528752 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.623984 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.744155 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:30.758067 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:30.762471 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.771698 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.835217 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:30.857784 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:30.860066 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:30.871788 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:30.897102 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:30.929749 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:31.085582 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:31.101464 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:31.128122 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:31.131269 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:31.142207 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:31.142388 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:15:31.164422 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:31.167690 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:31.195494 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:31.226758 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:31.249502 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:31.274339 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:31.397138 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:31.449113 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:15:31.572396 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:31.681691 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:31.683436 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:31.685070 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:31.698752 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:31.723430 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:31.728866 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:31.729046 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:31.802021 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:31.824934 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:31.830482 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:31.835867 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:31.843133 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:31.847339 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:31.856626 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:31.864547 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:31.885896 23912 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.134:42431, user_credentials={real_user=slave}} blocked reactor thread for 57008.4us
W20240501 20:15:31.919384 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:32.015110 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:32.019775 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:32.076375 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:32.079344 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:32.111974 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:32.114249 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:32.187386 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:32.208546 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:32.308344 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:32.321169 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:15:32.325958 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:32.336290 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:32.345041 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:32.359206 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:32.365981 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:15:32.368084 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:32.388481 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:32.432624 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:32.439710 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:32.470533 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:32.491904 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:32.492051 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:32.519886 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:32.567754 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:32.603370 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:32.628221 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:32.636101 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:15:32.655473 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:32.711035 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:32.754721 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:32.783056 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:33.013401 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:33.015436 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:33.021509 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:33.040344 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:33.045302 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:33.053792 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:33.055958 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:33.091929 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:33.092834 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:33.093899 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:33.133051 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:33.174952 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:33.183717 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:33.230716 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:33.242734 23782 consensus_peers.cc:485] T 0722c64801734730abefe2a742da549e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:33.253183 21294 consensus_peers.cc:485] T 62e27bcc1cf34d8ab5d50331bee12886 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:33.321707 21294 consensus_peers.cc:485] T 57cda4be1d824e629b2938d44c5f2398 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:33.356571 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:33.391688 23782 consensus_peers.cc:485] T b472171e77e144ffa8fa4cdbd40a03a0 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:33.528457 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:33.544034 21294 consensus_peers.cc:485] T 236ba60733e1416395346179be3b3888 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:33.623533 21034 consensus_peers.cc:485] T 170ccc5b4ec949f1aa0652e43a3b9e5c P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:33.625007 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 20:15:33.628031 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:33.655743 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:33.658629 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:33.682626 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:33.710966 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:33.743657 21034 consensus_peers.cc:485] T d25f96315839453d8cd5458417404e34 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:33.745208 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:33.777463 23782 consensus_peers.cc:485] T 7a23f60c87794a49ae1f82dff06fe881 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:33.797344 21294 consensus_peers.cc:485] T b4c4664b17f04e2d8285b7851b27d1a1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:33.848592 21294 consensus_peers.cc:485] T 24cc6ff1a0104ac690c8d8f82c17200c P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:33.967519 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:34.033129 21294 consensus_peers.cc:485] T aefb41ceeb524a28a6c17db6b11316e2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:15:34.116202 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:34.127316 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:34.137949 20903 consensus_peers.cc:485] T 73fcc9ac412643088c996b76fbb1e680 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:34.143056 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:34.189502 20903 consensus_peers.cc:485] T 30d2aada073b4d8989d7db8be4034756 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:34.259815 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 20:15:34.286454 24046 consensus_peers.cc:485] T d3c0aa45580749a4bd6fa7ccd635cb38 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:34.287627 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:34.304737 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 20:15:34.316746 21294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:15:34.318145 21294 consensus_peers.cc:485] T 4587a9d5392742b58ec8d64619f2dda1 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:34.341709 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:34.347858 23782 consensus_peers.cc:485] T 224f83ad3d514a7fa5cdfa4e70ff2fd8 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:34.357926 21034 consensus_peers.cc:485] T e1476e8f4c9343928f68365fd2772f0f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:34.492964 21294 consensus_peers.cc:485] T ed9b47caf0a041fd962e4f18b0de5ff2 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:34.502162 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:34.502183 21294 consensus_peers.cc:485] T c1e295a46a174f3f8673e03f57890b58 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:34.556631 24046 consensus_peers.cc:485] T 3f3951bb18524e41aa6836122e97e7f5 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:34.559990 23782 consensus_peers.cc:485] T cf1ac1f2f3d642e9a9c785bb4963af1f P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:34.640736 21034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:15:34.643270 21034 consensus_peers.cc:485] T a115fe66d11143ea8e24aa891e4e2f33 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:34.647959 21294 consensus_peers.cc:485] T 43e80df9af0e40c5acc4dea95d7e95a8 P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:34.689700 21034 consensus_peers.cc:485] T 8d75b25758954877ab36e8029fb29463 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:34.695101 24046 consensus_peers.cc:485] T ed53558d93214d34aedca8505870ec61 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:34.733963 21034 consensus_peers.cc:485] T 3a7e956aedc44f56b3bf629c80ff65e4 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:34.734498 20903 consensus_peers.cc:485] T 538f2f3892624f63bc88064e5ecd9443 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:34.777024 23647 consensus_peers.cc:485] T 0fda032f2be744fe95170b3c3287d6ae P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:34.811478 21034 consensus_peers.cc:485] T 6c3bdb74d3d140e397ce0b5bdef5361d P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:34.836800 23782 consensus_peers.cc:485] T 021fed14d5a140f3b42d359bae264f69 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:34.887952 23647 consensus_peers.cc:485] T 4715a768081a49228ab47b8071c40c8f P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:15:34.897677 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:15:34.903630 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:34.906812 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:34.935317 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:34.949793 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
W20240501 20:15:34.950541 23782 consensus_peers.cc:485] T a7498fc5cf5144d58712eb2c8f188c5e P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:15:34.957021 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:34.967103 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 20:15:34.985020 24046 consensus_peers.cc:485] T 4341f17784bc45359a9648ab47c39628 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:15:34.993534 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:35.008131 24046 consensus_peers.cc:485] T 718853c0da3d4a68ab478a6e79cdd438 P 8559229e044b4e1ea661b881b3f9af50 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:35.028378 21034 consensus_peers.cc:485] T c58dd83ebd0949dea6f4567067c8dfc9 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:35.037348 20903 consensus_peers.cc:485] T 10a4d24c5c844c8c9ccd191b1a2b0b00 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:35.037670 23910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.251.132:46399, user_credentials={real_user=slave}} blocked reactor thread for 62625.3us
W20240501 20:15:35.047748 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240501 20:15:35.097800 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:15:35.208137 21034 consensus_peers.cc:485] T d75d2ef39a5347e2b6fbe3145c207280 P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:35.213256 23782 consensus_peers.cc:485] T 6e3c1d24969140d2a5996ee27aead916 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:35.228439 23647 consensus_peers.cc:485] T d4ffd0f0fb874a809876af2a78ad07ab P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:15:35.291893 21034 consensus_peers.cc:485] T e1be0c0baa5e4c95915b93f659c4759f P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:35.347116 23782 consensus_peers.cc:485] T 2b57a80759944301a033140e482aec36 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:15:35.361222 21034 consensus_peers.cc:462] T a1cbbb090cb34465b6dac9c35cdba0dc P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111)
I20240501 20:15:35.412081 23691 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:35.420056 21338 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240501 20:15:35.436144 20903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:15:35.440647 23957 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:15:35.448666 20903 consensus_peers.cc:485] T c018d1b99ad749e59a2cf338001d38e4 P cda19d6a81c0473db302535ce2ada7f0 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:15:35.493258 23825 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:15:35.497781 24089 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:15:35.535094 21077 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:15:35.599892 20945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 20:15:35.625583 23782 consensus_peers.cc:485] T 420b46edf5fc438fb01e822440bce8f1 P c6dcaa8efff642dcb7c2c6e8be3a65f7 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:35.698357 23647 consensus_peers.cc:485] T 3d1c68b0b2184d75948c53381f5f0eb6 P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:15:35.702164 21034 consensus_peers.cc:485] T b70e144824074927bb0f4eafcdeadcec P feac73db0a3b4aafa374ca5dfd322b08 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:15:35.721946 23647 consensus_peers.cc:485] T 46677b6d96bf45d98d061e572a85ebdc P 51bf640e895b4e5eb91bdf7509ea7e46 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:15:35.728006 21294 consensus_peers.cc:485] T 748b4fd500f84a359bd8ee05b861300d P 61380023001a4ee9ada9ec2c6c1b7159 -> Peer 4af096f5a6a9440a94f6682b399efe20 (127.13.251.131:38349): Couldn't send request to peer 4af096f5a6a9440a94f6682b399efe20. Status: Network error: Client connection negotiation failed: client connection to 127.13.251.131:38349: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:15:35.754173 21294 consensus_peers.cc:485] T 2b4384496d9048259bf353bd58a15008 P 61380023001a4ee9ada9ec2c6c1b7159 ->